Pass custom metadata filename to Image/Audio folders
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5635). All of your documentation changes will be reflected on that endpoint.
2023-03-14T15:12:08Z
https://github.com/huggingface/datasets/pull/5635#issuecomment-1468292229
https://api.github.com/repos/huggingface/datasets/issues/5635
IC_kwDODunzps5XhFyF
https://api.github.com/repos/huggingface/datasets/issues/comments/1468292229/reactions
2023-03-14T15:12:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1468292229
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
COLLABORATOR
I'm not a big fan of this new param - I find assigning metadata files to splits via the `data_files` param cleaner. Also, assuming that the metadata filename is `metadata.json`/`metadata.csv` (I don't think we should allow other names), a user can do `load_dataset("imagefolder", data_dir="data")` to load a dataset with that structure.
2023-03-22T15:12:14Z
https://github.com/huggingface/datasets/pull/5635#issuecomment-1479753325
https://api.github.com/repos/huggingface/datasets/issues/5635
IC_kwDODunzps5YMz5t
https://api.github.com/repos/huggingface/datasets/issues/comments/1479753325/reactions
2023-03-22T15:12:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1479753325
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
@mariosasko  I don't really like this change in it's current state either but passing specific files with `data_files` also looks not quite user-friendly to me. The idea of providing specific parameter for metadata filename seems natural to me but I don't see a way for implementing it without some ugly changes in `load.py` (passing the param to factories and creating metadata patterns on the fly). Why don't you like this parameter?

For context: this PR emerged from the case where users wanted to use different metadata files with the same large set of images without copying directories on disk and it's not possible with `data_files` approach.

edit: ah no, it's possible if one  puts metadata files in different subdirs (so that the filenames can be left the same)
2023-03-22T17:08:51Z
https://github.com/huggingface/datasets/pull/5635#issuecomment-1479952296
https://api.github.com/repos/huggingface/datasets/issues/5635
IC_kwDODunzps5YNkeo
https://api.github.com/repos/huggingface/datasets/issues/comments/1479952296/reactions
2023-03-22T17:25:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1479952296
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
COLLABORATOR
>For context: this PR emerged from the case where users wanted to use different metadata files with the same large set of images without copying directories on disk and it's not possible with data_files approach.
>
>edit: ah no, it's possible if one puts metadata files in different subdirs (so that the filenames can be left the same)

Seems low prio, but one way to address this would be by allowing to pass  "exclude patterns" to `data_files`
2023-03-22T17:50:31Z
https://github.com/huggingface/datasets/pull/5635#issuecomment-1480011983
https://api.github.com/repos/huggingface/datasets/issues/5635
IC_kwDODunzps5YNzDP
https://api.github.com/repos/huggingface/datasets/issues/comments/1480011983/reactions
2023-03-22T17:50:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1480011983
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
This is a quick fix. 
Now it requires to pass data via `data_files` parameters and include a required metadata file there and pass its filename as `metadata_filename` parameter.
For example, with the structure like:
```
data
   images_dir/
      im1.jpg
      im2.jpg
      ...
   metadata_dir/
      meta_file1.jsonl
      meta_file2.jsonl
      ...
```
to load data with `metadata_file1.jsonl` do:
```python
ds = load_dataset("imagefolder", data_files=["data/images_dir/**", "data/metadata_dir/meta_file1.jsonl"], metadata_filename="meta_file1.jsonl")
```

Note that if you have multiple splits, metadata file should be specified in each of them in `data_files`, smth like:
```python
data_files={
"train": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"], 
"test": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"]
}
```