wmt download speed example
MEMBER
Thanks for creating the issue :)
The download link for wmt-en-de raw looks like a mirror. We should use that instead of the current url.
Is this mirror official ?

Also it looks like for `ro-en` it tried to download other languages. If we manage to only download the one that is asked it'd be cool

Also cc @patrickvonplaten 
2020-08-24T10:33:22Z
https://github.com/huggingface/datasets/issues/525#issuecomment-679047729
https://api.github.com/repos/huggingface/datasets/issues/525
MDEyOklzc3VlQ29tbWVudDY3OTA0NzcyOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/679047729/reactions
2020-08-24T10:33:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/679047729
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Mirror is not official.
2020-08-27T01:46:12Z
https://github.com/huggingface/datasets/issues/525#issuecomment-681268969
https://api.github.com/repos/huggingface/datasets/issues/525
MDEyOklzc3VlQ29tbWVudDY4MTI2ODk2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/681268969/reactions
2020-08-27T01:46:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/681268969
https://avatars.githubusercontent.com/u/6045025?v=4
https://api.github.com/users/sshleifer/events{/privacy}
https://api.github.com/users/sshleifer/followers
https://api.github.com/users/sshleifer/following{/other_user}
https://api.github.com/users/sshleifer/gists{/gist_id}
https://github.com/sshleifer
sshleifer
MDQ6VXNlcjYwNDUwMjU=
https://api.github.com/users/sshleifer/orgs
https://api.github.com/users/sshleifer/received_events
https://api.github.com/users/sshleifer/repos
https://api.github.com/users/sshleifer/starred{/owner}{/repo}
https://api.github.com/users/sshleifer/subscriptions
User
https://api.github.com/users/sshleifer
public
MEMBER
Shall we host the files ourselves or it is fine to use this mirror in your opinion ?
2020-08-27T08:14:13Z
https://github.com/huggingface/datasets/issues/525#issuecomment-681756285
https://api.github.com/repos/huggingface/datasets/issues/525
MDEyOklzc3VlQ29tbWVudDY4MTc1NjI4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/681756285/reactions
2020-08-27T08:14:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/681756285
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Should we add an argument in `load_dataset` to override some URL with a custom URL (e.g. mirror) or a local path?

This could also be used to provide local files instead of the original files as requested by some users (e.g. when you made a dataset with the same format than SQuAD and what to use it instead of the official dataset files).
2020-08-27T08:35:16Z
https://github.com/huggingface/datasets/issues/525#issuecomment-681788590
https://api.github.com/repos/huggingface/datasets/issues/525
MDEyOklzc3VlQ29tbWVudDY4MTc4ODU5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/681788590/reactions
2020-08-27T08:35:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/681788590
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
@lhoestq I think we should host it ourselves. I'll put the subset of wmt (without preprocessed files) that we need on s3 and post a link over the weekend.
2020-08-27T22:03:58Z
https://github.com/huggingface/datasets/issues/525#issuecomment-682213991
https://api.github.com/repos/huggingface/datasets/issues/525
MDEyOklzc3VlQ29tbWVudDY4MjIxMzk5MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/682213991/reactions
2020-08-27T22:07:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/682213991
https://avatars.githubusercontent.com/u/6045025?v=4
https://api.github.com/users/sshleifer/events{/privacy}
https://api.github.com/users/sshleifer/followers
https://api.github.com/users/sshleifer/following{/other_user}
https://api.github.com/users/sshleifer/gists{/gist_id}
https://github.com/sshleifer
sshleifer
MDQ6VXNlcjYwNDUwMjU=
https://api.github.com/users/sshleifer/orgs
https://api.github.com/users/sshleifer/received_events
https://api.github.com/users/sshleifer/repos
https://api.github.com/users/sshleifer/starred{/owner}{/repo}
https://api.github.com/users/sshleifer/subscriptions
User
https://api.github.com/users/sshleifer
public
NONE
Is there a solution yet? The download speed is still too slow. 60-70kbps download for wmt16 and around 100kbps for wmt19. @sshleifer 
2021-02-18T16:14:44Z
https://github.com/huggingface/datasets/issues/525#issuecomment-781457524
https://api.github.com/repos/huggingface/datasets/issues/525
MDEyOklzc3VlQ29tbWVudDc4MTQ1NzUyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/781457524/reactions
2021-02-18T16:15:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/781457524
https://avatars.githubusercontent.com/u/16959771?v=4
https://api.github.com/users/Rachneet/events{/privacy}
https://api.github.com/users/Rachneet/followers
https://api.github.com/users/Rachneet/following{/other_user}
https://api.github.com/users/Rachneet/gists{/gist_id}
https://github.com/Rachneet
Rachneet
MDQ6VXNlcjE2OTU5Nzcx
https://api.github.com/users/Rachneet/orgs
https://api.github.com/users/Rachneet/received_events
https://api.github.com/users/Rachneet/repos
https://api.github.com/users/Rachneet/starred{/owner}{/repo}
https://api.github.com/users/Rachneet/subscriptions
User
https://api.github.com/users/Rachneet
public
MEMBER
I'm working on mirror links which will provide high download speed :)
See https://github.com/huggingface/datasets/issues/1892
2021-02-18T16:16:07Z
https://github.com/huggingface/datasets/issues/525#issuecomment-781458539
https://api.github.com/repos/huggingface/datasets/issues/525
MDEyOklzc3VlQ29tbWVudDc4MTQ1ODUzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/781458539/reactions
2021-02-18T16:16:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/781458539
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Resolved via https://github.com/huggingface/datasets/pull/1912
2022-10-04T17:45:39Z
https://github.com/huggingface/datasets/issues/525#issuecomment-1267346971
https://api.github.com/repos/huggingface/datasets/issues/525
IC_kwDODunzps5Lii4b
https://api.github.com/repos/huggingface/datasets/issues/comments/1267346971/reactions
2022-10-04T17:45:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1267346971
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
Continuing from the slack 1.0 roadmap thread w @lhoestq , I realized the slow downloads is only a thing sometimes. Here are a few examples, I suspect there are multiple issues. All commands were run from the same gcp us-central-1f machine.

```
import nlp
nlp.load_dataset('wmt16', 'de-en')
```
Downloads at 49.1 KB/S

Whereas 
```
pip install gdown # download from google drive
!gdown https://drive.google.com/uc?id=1iO7um-HWoNoRKDtw27YUSgyeubn9uXqj
```
Downloads at 127 MB/s. (The file is a copy of wmt-en-de raw).


```
nlp.load_dataset('wmt16', 'ro-en')
```
goes at 27 MB/s, much faster. 

if we wget the same data from s3 is the same download speed, but ¼ the file size:
```
wget https://s3.amazonaws.com/datasets.huggingface.co/translation/wmt_en_ro_packed_200_rand.tgz
```

Finally,
```
nlp.load_dataset('wmt19', 'zh-en')
```
Starts fast, but broken. (duplicate of #493 )


