Add CC-100 dataset
MEMBER
Hello @lhoestq, I would like just to ask you if it is OK that I include this feature 9f32ba1 in this PR or you would prefer to have it in a separate one.

I was wondering whether include also a test, but I did not find any test for the other file formats...
2020-12-01T07:15:31Z
https://github.com/huggingface/datasets/pull/923#issuecomment-736273397
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDczNjI3MzM5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/736273397/reactions
2020-12-01T07:33:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/736273397
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Hi ! Sure that would be valuable to support .xz files. Feel free to open a separate PR for this.
And feel free to create the first test case for extracting compressed files if you have some inspiration (maybe create test_file_utils.py ?). We can still spend more time on tests next week when the sprint is over though so don't spend too much time on it.
2020-12-01T09:22:33Z
https://github.com/huggingface/datasets/pull/923#issuecomment-736338362
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDczNjMzODM2Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/736338362/reactions
2020-12-01T09:22:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/736338362
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq, DONE! ;) See PR #950.
2020-12-01T12:50:03Z
https://github.com/huggingface/datasets/pull/923#issuecomment-736531446
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDczNjUzMTQ0Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/736531446/reactions
2020-12-01T12:50:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/736531446
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Thanks for adding support for `.xz` files :)

Feel free to rebase from master to include it in your PR
2020-12-01T13:40:42Z
https://github.com/huggingface/datasets/pull/923#issuecomment-736558034
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDczNjU1ODAzNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/736558034/reactions
2020-12-01T13:40:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/736558034
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq DONE; I have merged instead, to avoid changing the history of my public PR ;)
2020-12-02T08:40:39Z
https://github.com/huggingface/datasets/pull/923#issuecomment-737080769
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDczNzA4MDc2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/737080769/reactions
2020-12-02T08:40:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/737080769
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Hi @lhoestq, I would need that you generate the dataset_infos.json and the dummy data for this dataset with a bigger computer. Sorry, but my laptop did not succeed...
2020-12-04T10:50:18Z
https://github.com/huggingface/datasets/pull/923#issuecomment-738714847
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDczODcxNDg0Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/738714847/reactions
2020-12-04T10:50:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/738714847
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Thanks for your work @albertvillanova 
We'll definitely look into it after this sprint :)
2020-12-04T13:36:27Z
https://github.com/huggingface/datasets/pull/923#issuecomment-738787427
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDczODc4NzQyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/738787427/reactions
2020-12-04T13:36:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/738787427
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Looks like #1456 added CC100 already.
The difference with your approach is that this implementation uses the `BuilderConfig` parameters to allow the creation of custom configs for all the languages, without having to specify them in the `BUILDER_CONFIGS` class attribute.
For example even if the dataset doesn't have a config for english already, you can still load the english CC100 with
```python
from datasets import load_dataset

load_dataset("cc100", lang="en")
```
2020-12-23T16:06:11Z
https://github.com/huggingface/datasets/pull/923#issuecomment-750365945
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDc1MDM2NTk0NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/750365945/reactions
2020-12-23T16:06:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750365945
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq, oops!! I remember having assigned this dataset to me in the Google sheet, besides having mentioned the corresponding issue in the Pull Request... Nevermind! :)
2020-12-23T16:17:38Z
https://github.com/huggingface/datasets/pull/923#issuecomment-750370328
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDc1MDM3MDMyOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/750370328/reactions
2020-12-23T16:17:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750370328
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Yes indeed I can see that...
Sorry for noticing that only now 

The code of the other PR ended up being pretty close to yours though
If you want to add more details to the cc100 dataset card or in the script feel to do so, any addition is welcome
2020-12-23T16:24:49Z
https://github.com/huggingface/datasets/pull/923#issuecomment-750373003
https://api.github.com/repos/huggingface/datasets/issues/923
MDEyOklzc3VlQ29tbWVudDc1MDM3MzAwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/750373003/reactions
2020-12-23T16:24:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750373003
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Add CC-100.

Close #773 