request to mirror wmt datasets, as they are really slow to download
MEMBER
Yes that would be awesome. Not only the download speeds are awful, but also some files are missing.
We list all the URLs in the datasets/wmt19/wmt_utils.py so we can make a script to download them all and host on S3.
Also I think most of the materials are under the CC BY-NC-SA 3.0 license (must double check) so it should be possible to redistribute the data with no issues.

cc @patrickvonplaten who knows more about the wmt scripts
2021-02-17T14:57:28Z
https://github.com/huggingface/datasets/issues/1892#issuecomment-780612835
https://api.github.com/repos/huggingface/datasets/issues/1892
MDEyOklzc3VlQ29tbWVudDc4MDYxMjgzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/780612835/reactions
2021-02-17T14:57:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/780612835
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Yeah, the scripts are pretty ugly! A big refactor would make sense here...and I also remember that the datasets were veeery slow to download
2021-02-17T15:22:41Z
https://github.com/huggingface/datasets/issues/1892#issuecomment-780630250
https://api.github.com/repos/huggingface/datasets/issues/1892
MDEyOklzc3VlQ29tbWVudDc4MDYzMDI1MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/780630250/reactions
2021-02-17T15:22:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/780630250
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
I'm downloading them.
I'm starting with the ones hosted on http://data.statmt.org which are the slowest ones
2021-02-18T09:54:58Z
https://github.com/huggingface/datasets/issues/1892#issuecomment-781224686
https://api.github.com/repos/huggingface/datasets/issues/1892
MDEyOklzc3VlQ29tbWVudDc4MTIyNDY4Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/781224686/reactions
2021-02-18T09:54:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/781224686
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq better to use our new git-based system than just raw S3, no? (that way we have built-in CDN etc.)
2021-02-18T13:55:43Z
https://github.com/huggingface/datasets/issues/1892#issuecomment-781360110
https://api.github.com/repos/huggingface/datasets/issues/1892
MDEyOklzc3VlQ29tbWVudDc4MTM2MDExMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/781360110/reactions
2021-02-18T13:55:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/781360110
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
MEMBER
Closing since the urls were changed to mirror urls in #1912 
2021-03-25T11:53:23Z
https://github.com/huggingface/datasets/issues/1892#issuecomment-806593544
https://api.github.com/repos/huggingface/datasets/issues/1892
MDEyOklzc3VlQ29tbWVudDgwNjU5MzU0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/806593544/reactions
2021-03-25T11:53:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/806593544
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi there! What about mirroring other datasets like [CCAligned](http://www.statmt.org/cc-aligned/) as well? All of them are really slow to download...
2021-10-26T06:55:41Z
https://github.com/huggingface/datasets/issues/1892#issuecomment-951616897
https://api.github.com/repos/huggingface/datasets/issues/1892
IC_kwDODunzps44uIWB
https://api.github.com/repos/huggingface/datasets/issues/comments/951616897/reactions
2021-10-26T06:55:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/951616897
https://avatars.githubusercontent.com/u/46803821?v=4
https://api.github.com/users/PrettyMeng/events{/privacy}
https://api.github.com/users/PrettyMeng/followers
https://api.github.com/users/PrettyMeng/following{/other_user}
https://api.github.com/users/PrettyMeng/gists{/gist_id}
https://github.com/PrettyMeng
PrettyMeng
MDQ6VXNlcjQ2ODAzODIx
https://api.github.com/users/PrettyMeng/orgs
https://api.github.com/users/PrettyMeng/received_events
https://api.github.com/users/PrettyMeng/repos
https://api.github.com/users/PrettyMeng/starred{/owner}{/repo}
https://api.github.com/users/PrettyMeng/subscriptions
User
https://api.github.com/users/PrettyMeng
public
Would it be possible to mirror the wmt data files under hf? Some of them take hours to download and not because of the local speed. They are all quite small datasets, just extremely slow to download.

Thank you!