[Large datasets] Speed up download and processing
MEMBER
Looks all good :)
I rebased from master and added a test for parallel `map_nested`
2020-09-08T13:12:29Z
https://github.com/huggingface/datasets/pull/563#issuecomment-688856507
https://api.github.com/repos/huggingface/datasets/issues/563
MDEyOklzc3VlQ29tbWVudDY4ODg1NjUwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/688856507/reactions
2020-09-08T13:12:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688856507
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
you're da best
2020-09-08T13:19:02Z
https://github.com/huggingface/datasets/pull/563#issuecomment-688860310
https://api.github.com/repos/huggingface/datasets/issues/563
MDEyOklzc3VlQ29tbWVudDY4ODg2MDMxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/688860310/reactions
2020-09-08T13:19:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688860310
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
Various improvements to speed-up creation and processing of large scale datasets.

Currently:
- distributed downloads
- remove etag from datafiles hashes to spare a request when restarting a failed download