Add CelebA Dataset
MEMBER
Hi @gchhablani! Thanks for all the contributions! We definitely want more image datasets, but Face datasets are tricky in general, in this one includes predicting attributes such as Attractiveness, Gender, or Race, which can be pretty problematic.

Would you be up for starting with only object classification or object detection datasets instead? (Your CIFAR-100 contribution will be super useful for example!)
2021-02-05T20:47:28Z
https://github.com/huggingface/datasets/pull/1828#issuecomment-774280131
https://api.github.com/repos/huggingface/datasets/issues/1828
MDEyOklzc3VlQ29tbWVudDc3NDI4MDEzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/774280131/reactions
2021-02-05T20:49:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774280131
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
CONTRIBUTOR
Hi @yjernite, You're welcome. I am enjoying adding new datasets :)
By "pretty problematic", are you referring to the ethical issues? I used TFDS's [CelebA](https://github.com/tensorflow/datasets/blob/5ef7861470896acb6f74dacba85036001e4f1b8c/tensorflow_datasets/image/celeba.py#L91) as a reference. Here they mention in a "Note" that CelebA  "may contain potential bias". Can we not do the same? I skipped the note for now, and we can add it. However, if you feel this isn't the right time, then I won't pursue this further. 

But, can this issue be handled at a later stage? Does this also apply for my Hateful Memes Issue #1810?

Also, how can I 
1. load a part of the dataset? since `load_dataset(<>,split='train[10:20]')` still loads all the examples.
2. make `datasets_infos.json` for huge datasets which have a single configuration?

I will ofcourse be looking for other datasets to add regardless. 

2021-02-05T21:07:34Z
https://github.com/huggingface/datasets/pull/1828#issuecomment-774289126
https://api.github.com/repos/huggingface/datasets/issues/1828
MDEyOklzc3VlQ29tbWVudDc3NDI4OTEyNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/774289126/reactions
2021-02-05T21:31:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774289126
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
MEMBER
It's definitely a thorny question. The short answer is: Hateful Memes and hate speech detection datasets are different since their use case is specifically to train systems to identify and hopefully remove hateful content, whereas the purpose of a dataset that has an Attractiveness score as output is implicitly to train more models to rate "Attractiveness". 

As far as warning about the "potential biases", I do not think it is quite enough, especially because it is hard to guarantee that every potential user will read the documentation (it is also an insufficient warning.)

Note that we do have higher standards for the dataset cards of hate speech and hateful memes datasets, so if you do choose to add that one yourself we will ask that you summarize the relevant literature in the Social Impact section.

If you really need to add this dataset for your own research for the explicit purpose of studying these biases, you can add it as a community provided dataset following https://huggingface.co/docs/datasets/master/share_dataset.html#sharing-a-community-provided-dataset but I'd recommend just skipping it for now.
2021-02-05T22:06:13Z
https://github.com/huggingface/datasets/pull/1828#issuecomment-774313905
https://api.github.com/repos/huggingface/datasets/issues/1828
MDEyOklzc3VlQ29tbWVudDc3NDMxMzkwNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/774313905/reactions
2021-02-05T22:07:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774313905
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
MEMBER
So currently you do need to download the whole dataset when using it, we are working on making it easier to stream parts of it from a remote host. You can also use the filesystem integration if local storage is an issue:
https://huggingface.co/docs/datasets/master/filesystems.html

2021-02-05T22:10:46Z
https://github.com/huggingface/datasets/pull/1828#issuecomment-774315652
https://api.github.com/repos/huggingface/datasets/issues/1828
MDEyOklzc3VlQ29tbWVudDc3NDMxNTY1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/774315652/reactions
2021-02-05T22:10:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774315652
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
MEMBER
I don't think we have a great solution for `dataset_infos.json` with a single very large config when storage space is an issue, but it should be solved by the same upcoming feature mentioned above
2021-02-05T22:11:42Z
https://github.com/huggingface/datasets/pull/1828#issuecomment-774316061
https://api.github.com/repos/huggingface/datasets/issues/1828
MDEyOklzc3VlQ29tbWVudDc3NDMxNjA2MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/774316061/reactions
2021-02-05T22:11:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774316061
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
CONTRIBUTOR
Okay, then I won't pursue this one further. I'll keep this branch on my repository just in case the possibility of adding this dataset comes up in the future.

> So currently you do need to download the whole dataset when using it, we are working on making it easier to stream parts of it from a remote host. You can also use the filesystem integration if local storage is an issue:
> https://huggingface.co/docs/datasets/master/filesystems.html

After downloading the whole dataset (around 1.4GB), it still loads all the examples despite using `split='train[:10%]'` or `split='train[10:20]'`. 

EDIT: I think this would happen only when the examples are generated for the first time and saved to the cache. Streaming parts of the data from a remote host sounds amazing! But, would that also allow for streaming examples of the data from the local cache? (without saving all the examples the first time).

What I used:
`d = load_dataset('./datasets/celeb_a',split='train[:10]')`
Output:
`570 examples [01:33,  6.25 examples/s]` and it keeps going. 

EDIT 2: After a few thousand images, I get the following error:
```python
OSError: [Errno 24] Too many open files: '~/.cache/huggingface/datasets/celeb_a/default/1.1.0/01f9dca66039ab7c40b91b09af47a5fa8c3e49dc8d55df50da55b14116229207.incomplete'
```
I understand this is because of the way I load the images :
```python
Image.open(<path>)
```
What could be better alternative? I am only asking in case I face the same issues in the future.
2021-02-06T05:27:56Z
https://github.com/huggingface/datasets/pull/1828#issuecomment-774403267
https://api.github.com/repos/huggingface/datasets/issues/1828
MDEyOklzc3VlQ29tbWVudDc3NDQwMzI2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/774403267/reactions
2021-02-06T14:03:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774403267
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
MEMBER
Just some addition about loading only a subset of the data:
Currently if even you specify `split='train[:10]'`, it downloads and generate the full dataset, so that you can pick another part afterward if you want to. We may change that in the future and use streaming.

And about your open files issue, you can try to close each image file after reading its content.
2021-02-10T15:11:40Z
https://github.com/huggingface/datasets/pull/1828#issuecomment-776774269
https://api.github.com/repos/huggingface/datasets/issues/1828
MDEyOklzc3VlQ29tbWVudDc3Njc3NDI2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/776774269/reactions
2021-02-10T15:11:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/776774269
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq,
Thanks for your response.

I used `gc.collect()` inside the loop and that worked for me. I think since we are using a generator, and if I have something like `train[100000:100002]`, we will need to generate the first 1000001 examples and store. Ofcourse, this feature isn't a necessity right now, I suppose.
2021-02-11T17:00:11Z
https://github.com/huggingface/datasets/pull/1828#issuecomment-777642049
https://api.github.com/repos/huggingface/datasets/issues/1828
MDEyOklzc3VlQ29tbWVudDc3NzY0MjA0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/777642049/reactions
2021-02-11T17:00:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/777642049
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
CONTRIBUTOR
Closing this PR.
2021-02-18T14:17:07Z
https://github.com/huggingface/datasets/pull/1828#issuecomment-781374216
https://api.github.com/repos/huggingface/datasets/issues/1828
MDEyOklzc3VlQ29tbWVudDc4MTM3NDIxNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/781374216/reactions
2021-02-18T14:17:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/781374216
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
Trying to add CelebA Dataset. 
Need help with testing. Loading examples takes a lot of time so I am unable to generate the `dataset_infos.json` and unable to test. Also, need help with creating `dummy_data.zip`.

Additionally, trying to load a few examples using `load_dataset('./datasets/celeb_a',split='train[10:20]')` still loads all the examples (doesn't stop at 10).