Add german common crawl dataset
MEMBER
Hi @Phil1108 !
Have you had a chance to take a look at my suggestions ?
Feel free to ping me if you have questions or if you're ready for a review

Thanks again for adding this dataset, this one is very useful !
2020-12-22T17:36:26Z
https://github.com/huggingface/datasets/pull/1425#issuecomment-749679388
https://api.github.com/repos/huggingface/datasets/issues/1425
MDEyOklzc3VlQ29tbWVudDc0OTY3OTM4OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/749679388/reactions
2020-12-22T17:36:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749679388
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> 
> 
> Hi @Phil1108 !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or if you're ready for a review
> 
> Thanks again for adding this dataset, this one is very useful !

Hello @lhoestq , 

okay so should we go on with only the 2 parts or wait until the hosting of all 200 GB is cleared at HF? 
If you want to merge it with the 2 parts only at first und upgrade later to all, sure I'll include the minor changes in the next few days and update it here
2020-12-22T19:01:37Z
https://github.com/huggingface/datasets/pull/1425#issuecomment-749720601
https://api.github.com/repos/huggingface/datasets/issues/1425
MDEyOklzc3VlQ29tbWVudDc0OTcyMDYwMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/749720601/reactions
2020-12-22T19:01:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749720601
https://avatars.githubusercontent.com/u/39518904?v=4
https://api.github.com/users/Phil1108/events{/privacy}
https://api.github.com/users/Phil1108/followers
https://api.github.com/users/Phil1108/following{/other_user}
https://api.github.com/users/Phil1108/gists{/gist_id}
https://github.com/Phil1108
Phil1108
MDQ6VXNlcjM5NTE4OTA0
https://api.github.com/users/Phil1108/orgs
https://api.github.com/users/Phil1108/received_events
https://api.github.com/users/Phil1108/repos
https://api.github.com/users/Phil1108/starred{/owner}{/repo}
https://api.github.com/users/Phil1108/subscriptions
User
https://api.github.com/users/Phil1108
public
MEMBER
I think we can wait a bit to find a solution for the hosting and update the PR to include all the files when it's done.
In the meantime you can update the PR to take into account the suggestions regarding the dataset card, the parallel downloads and the use of eval.

A safer alternative to eval is `ast.literal_eval`, it should do the job
2020-12-23T09:59:47Z
https://github.com/huggingface/datasets/pull/1425#issuecomment-750064297
https://api.github.com/repos/huggingface/datasets/issues/1425
MDEyOklzc3VlQ29tbWVudDc1MDA2NDI5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/750064297/reactions
2020-12-23T09:59:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750064297
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Thanks for your contribution, @Phil1108. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
2022-09-30T14:21:12Z
https://github.com/huggingface/datasets/pull/1425#issuecomment-1263640663
https://api.github.com/repos/huggingface/datasets/issues/1425
IC_kwDODunzps5LUaBX
https://api.github.com/repos/huggingface/datasets/issues/comments/1263640663/reactions
2022-09-30T14:21:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1263640663
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Adding a subpart of the Common Crawl which was extracted with this repo https://github.com/facebookresearch/cc_net and additionally filtered for duplicates 