Add MetaShift dataset
CONTRIBUTOR
I would like to take this up and give it a shot. Any image specific - dataset guidelines to keep in mind ? Thank you.
2022-03-08T14:59:56Z
https://github.com/huggingface/datasets/issues/3813#issuecomment-1061869500
https://api.github.com/repos/huggingface/datasets/issues/3813
IC_kwDODunzps4_Ste8
https://api.github.com/repos/huggingface/datasets/issues/comments/1061869500/reactions
2022-03-08T14:59:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1061869500
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
#self-assign
2022-03-08T15:13:46Z
https://github.com/huggingface/datasets/issues/3813#issuecomment-1061884107
https://api.github.com/repos/huggingface/datasets/issues/3813
IC_kwDODunzps4_SxDL
https://api.github.com/repos/huggingface/datasets/issues/comments/1061884107/reactions
2022-03-08T15:13:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1061884107
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
I've started working on adding this dataset. I require some inputs on the following : 

Ref for the initial draft [here](https://github.com/dnaveenr/datasets/blob/add_metashift_dataset/datasets/metashift/metashift.py)
1. The dataset does not have a typical - train/test/val split. What do we do for the _split_generators() function ? How do we go about this ?
2. This dataset builds on the Visual Genome dataset, using a metadata file. The dataset is generated using generate_full_MetaShift.py script. By default, the authors choose to generate the dataset only for a SELECTED_CLASSES. The following script is used : 
Code : https://github.com/Weixin-Liang/MetaShift/blob/main/dataset/generate_full_MetaShift.py 
Info : https://metashift.readthedocs.io/en/latest/sub_pages/download_MetaShift.html#generate-the-full-metashift-dataset
Can I just copy over the required functions into the metashift.py to generate the dataset ?
3. How do we complete the _generate_examples for this dataset ?

The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Inputs, suggestions would be helpful. Thank you.
2022-03-09T18:43:23Z
https://github.com/huggingface/datasets/issues/3813#issuecomment-1063242479
https://api.github.com/repos/huggingface/datasets/issues/3813
IC_kwDODunzps4_X8rv
https://api.github.com/repos/huggingface/datasets/issues/comments/1063242479/reactions
2022-03-09T18:43:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1063242479
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
I think @mariosasko and @lhoestq should be able to help here 😄 
2022-03-09T21:25:30Z
https://github.com/huggingface/datasets/issues/3813#issuecomment-1063386747
https://api.github.com/repos/huggingface/datasets/issues/3813
IC_kwDODunzps4_Yf57
https://api.github.com/repos/huggingface/datasets/issues/comments/1063386747/reactions
2022-03-09T21:25:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1063386747
https://avatars.githubusercontent.com/u/7246357?v=4
https://api.github.com/users/osanseviero/events{/privacy}
https://api.github.com/users/osanseviero/followers
https://api.github.com/users/osanseviero/following{/other_user}
https://api.github.com/users/osanseviero/gists{/gist_id}
https://github.com/osanseviero
osanseviero
MDQ6VXNlcjcyNDYzNTc=
https://api.github.com/users/osanseviero/orgs
https://api.github.com/users/osanseviero/received_events
https://api.github.com/users/osanseviero/repos
https://api.github.com/users/osanseviero/starred{/owner}{/repo}
https://api.github.com/users/osanseviero/subscriptions
User
https://api.github.com/users/osanseviero
public
MEMBER
Hi ! Thanks for adding this dataset :) Let me answer your questions:

1. in this case you can put everything in the "train" split
2. Yes you can copy the script (provided you also include the MIT license of the code in the file header for example). Though we ideally try to not create new directories nor files when generating dataset, so if possible this script should be adapted to not create the file structure they mentioned, but instead yield the images one by one in `_generate_examples`. Let me know if you think this is feasible
3. see point 2 haha

> The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Yup ! We can also define a `selected_classes` parameter such that users can do
```python
load_dataset("metashift", selected_classes=["cat", "dog", ...])
```
2022-03-10T11:55:37Z
https://github.com/huggingface/datasets/issues/3813#issuecomment-1063976865
https://api.github.com/repos/huggingface/datasets/issues/3813
IC_kwDODunzps4_av-h
https://api.github.com/repos/huggingface/datasets/issues/comments/1063976865/reactions
2022-03-10T11:55:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1063976865
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Great. This is helpful. Thanks @lhoestq .
Regarding Point 2, I'll try using yield instead of creating the directories and see if its feasible. selected_classes config sounds good.
2022-03-10T16:35:45Z
https://github.com/huggingface/datasets/issues/3813#issuecomment-1064264328
https://api.github.com/repos/huggingface/datasets/issues/3813
IC_kwDODunzps4_b2KI
https://api.github.com/repos/huggingface/datasets/issues/comments/1064264328/reactions
2022-03-10T16:35:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1064264328
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
COLLABORATOR
Closed via #3900 
2022-04-10T13:39:59Z
https://github.com/huggingface/datasets/issues/3813#issuecomment-1094277239
https://api.github.com/repos/huggingface/datasets/issues/3813
IC_kwDODunzps5BOVh3
https://api.github.com/repos/huggingface/datasets/issues/comments/1094277239/reactions
2022-04-10T13:39:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1094277239
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
