Add note about loading image / audio files to docs
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4925). All of your documentation changes will be reflected on that endpoint.
2022-09-02T10:42:20Z
https://github.com/huggingface/datasets/pull/4925#issuecomment-1235346429
https://api.github.com/repos/huggingface/datasets/issues/4925
IC_kwDODunzps5JoeP9
https://api.github.com/repos/huggingface/datasets/issues/comments/1235346429/reactions
2022-09-02T10:42:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1235346429
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Thanks for the feedback @polinaeterna ! I've reworded the docs a bit to integrate your comments and this should be ready for another review :)
2022-09-05T09:40:13Z
https://github.com/huggingface/datasets/pull/4925#issuecomment-1236771503
https://api.github.com/repos/huggingface/datasets/issues/4925
IC_kwDODunzps5Jt6Kv
https://api.github.com/repos/huggingface/datasets/issues/comments/1236771503/reactions
2022-09-05T09:40:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1236771503
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
> I've just realized that there is another PR about audio documentation open: #4872
> and there the more detailed description on how to use `audiofolder` is moved to another section ("Create an audio dataset")

Ah yes, let's add a comment to #4872 - that will be simpler than the alternatives :)
2022-09-07T10:41:55Z
https://github.com/huggingface/datasets/pull/4925#issuecomment-1239218947
https://api.github.com/repos/huggingface/datasets/issues/4925
IC_kwDODunzps5J3PsD
https://api.github.com/repos/huggingface/datasets/issues/comments/1239218947/reactions
2022-09-07T10:41:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1239218947
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
COLLABORATOR
@polinaeterna @lhoestq What do you think about adding support for the metadata format from Kaggle (one metadata file for each split with the name equal to the split name) to ImageFolder/AudioFolder? I also think we can relax some requirements a bit by:
* allowing `filename` as the name of the main metadata column (currently, only `file_path` is allowed)
* not requiring that the features of all the given metadata files are equal. Instead, we can have a soft check by using `_check_if_features_can_be_aligned` + `_align_features`. The rationale is that train/val metadata often has extra columns compared to test metadata.

These changes would allow us to load the Kaggle dataset linked in the forum thread without any "interventions".

PS: this metadata format for ImageFolder was also proposed by @abhishekkrthakur initially.

2022-09-14T17:08:06Z
https://github.com/huggingface/datasets/pull/4925#issuecomment-1247063717
https://api.github.com/repos/huggingface/datasets/issues/4925
IC_kwDODunzps5KVK6l
https://api.github.com/repos/huggingface/datasets/issues/comments/1247063717/reactions
2022-09-14T17:09:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1247063717
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Can you give more details about the Kaggle format ? I'm down to discuss it in a separate issue if you don't mind.

> allowing filename as the name of the main metadata column (currently, only file_path is allowed)

`filename` refers to the name of the file, so there's no logic about relative path or directories. If I recall correctly this is what we're doing right now so why not

> not requiring that the features of all the given metadata files are equal. Instead, we can have a soft check by using _check_if_features_can_be_aligned + _align_features. The rationale is that train/val metadata often has extra columns compared to test metadata.

+1 and we can set to None the missing features
2022-09-15T15:33:14Z
https://github.com/huggingface/datasets/pull/4925#issuecomment-1248269609
https://api.github.com/repos/huggingface/datasets/issues/4925
IC_kwDODunzps5KZxUp
https://api.github.com/repos/huggingface/datasets/issues/comments/1248269609/reactions
2022-09-15T15:33:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1248269609
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
I'm not sure if this is worth opening a new issue :).

What I mean by the Kaggle format is the structure like this one (the name of a metadata file is equal to the directory it "references"):
```
- train
    - img1.jpeg
    - img2.jpeg
    - ...
- test
    - img1.jpeg
    - img2.jpeg
    - ...   
- train.csv
- test.csv
```



2022-09-16T16:13:38Z
https://github.com/huggingface/datasets/pull/4925#issuecomment-1249548556
https://api.github.com/repos/huggingface/datasets/issues/4925
IC_kwDODunzps5KepkM
https://api.github.com/repos/huggingface/datasets/issues/comments/1249548556/reactions
2022-09-16T16:13:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1249548556
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Sounds nice !
2022-09-16T16:20:31Z
https://github.com/huggingface/datasets/pull/4925#issuecomment-1249554652
https://api.github.com/repos/huggingface/datasets/issues/4925
IC_kwDODunzps5KerDc
https://api.github.com/repos/huggingface/datasets/issues/comments/1249554652/reactions
2022-09-16T16:20:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1249554652
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@mariosasko +1 to allowing different features set and metadata filenames corresponding to split names

Considering filename column - right now it's even called `file_name` now, which is not nice because in fact it's a relative file path indeed, so I think it should be `file_path` (and I don't know why I haven't thought about it before the release...)
2022-09-23T13:52:04Z
https://github.com/huggingface/datasets/pull/4925#issuecomment-1256245312
https://api.github.com/repos/huggingface/datasets/issues/4925
IC_kwDODunzps5K4MhA
https://api.github.com/repos/huggingface/datasets/issues/comments/1256245312/reactions
2022-09-23T13:52:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1256245312
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
@lewtun don't you mind if I close this pull request as I've integrated your changes in https://github.com/huggingface/datasets/pull/4872 ? (it doesn't have a link to a kaggle example though)
2022-09-23T13:59:07Z
https://github.com/huggingface/datasets/pull/4925#issuecomment-1256253145
https://api.github.com/repos/huggingface/datasets/issues/4925
IC_kwDODunzps5K4ObZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1256253145/reactions
2022-09-23T13:59:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1256253145
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
This PR adds a small note about how to load image / audio datasets that have multiple splits in their dataset structure.

Related forum thread: https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447

cc @NielsRogge 