create our own DownloadManager
CONTRIBUTOR
Looks great to me! 
2020-04-24T13:24:48Z
https://github.com/huggingface/datasets/pull/16#issuecomment-619007632
https://api.github.com/repos/huggingface/datasets/issues/16
MDEyOklzc3VlQ29tbWVudDYxOTAwNzYzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/619007632/reactions
2020-04-24T13:24:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/619007632
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
The new download manager is ready. I removed the old folder and I fixed a few remaining dependencies.
I tested it on squad and a few others from the dataset folder and it works fine.

The only impact of these changes is that it breaks the `download_and_prepare` script that was used to register the checksums when we create a dataset, as the checksum logic is not implemented.

Let me know if you have remarks
2020-04-24T15:46:02Z
https://github.com/huggingface/datasets/pull/16#issuecomment-619090994
https://api.github.com/repos/huggingface/datasets/issues/16
MDEyOklzc3VlQ29tbWVudDYxOTA5MDk5NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/619090994/reactions
2020-04-24T15:46:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/619090994
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Ok merged it (a bit fast for you to update the copyright, now I see that. but it's ok, we'll do a pass on these doc/copyright before releasing anyway)
2020-04-25T21:26:20Z
https://github.com/huggingface/datasets/pull/16#issuecomment-619442873
https://api.github.com/repos/huggingface/datasets/issues/16
MDEyOklzc3VlQ29tbWVudDYxOTQ0Mjg3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/619442873/reactions
2020-04-25T21:26:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/619442873
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Actually two additional things here @lhoestq (I merged too fast sorry, let's make a new PR for additional developments):
- I think we can remove some dependencies now (e.g. `promises`) in setup.py, can you have a look?
- also, I think we can remove the boto3 dependency like here: https://github.com/huggingface/transformers/pull/3968
2020-04-26T07:59:43Z
https://github.com/huggingface/datasets/pull/16#issuecomment-619505084
https://api.github.com/repos/huggingface/datasets/issues/16
MDEyOklzc3VlQ29tbWVudDYxOTUwNTA4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/619505084/reactions
2020-04-26T07:59:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/619505084
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
I tried to create our own - and way simpler - download manager, by replacing all the complicated stuff with our own `cached_path` solution.
With this implementation, I tried `dataset = nlp.load('squad')` and it seems to work fine.

For the implementation, what I did exactly:
- I copied the old download manager
- I removed all the dependences to the old `download` files
- I replaced all the download + extract calls by calls to `cached_path`
- I removed unused parameters (extract_dir, compute_stats) (maybe compute_stats could be re-added later if we want to compute stats...)
- I left some functions unimplemented for now. We will probably have to implement them because they are used by some datasets scripts (download_kaggle_data, iter_archive) or because we may need them at some point (download_checksums, _record_sizes_checksums)

Let me know if you think that this is going the right direction or if you have remarks.
Note: I didn't write any test yet as I wanted to read your remarks first