Support parallelized downloading in load_dataset with Spark
CONTRIBUTOR
Hi @lhoestq or other maintainers, this is ready for review, could you please take a look?
2023-05-05T01:04:01Z
https://github.com/huggingface/datasets/pull/5807#issuecomment-1535573119
https://api.github.com/repos/huggingface/datasets/issues/5807
IC_kwDODunzps5bhvx_
https://api.github.com/repos/huggingface/datasets/issues/comments/1535573119/reactions
2023-05-05T01:04:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1535573119
https://avatars.githubusercontent.com/u/12763339?v=4
https://api.github.com/users/es94129/events{/privacy}
https://api.github.com/users/es94129/followers
https://api.github.com/users/es94129/following{/other_user}
https://api.github.com/users/es94129/gists{/gist_id}
https://github.com/es94129
es94129
MDQ6VXNlcjEyNzYzMzM5
https://api.github.com/users/es94129/orgs
https://api.github.com/users/es94129/received_events
https://api.github.com/users/es94129/repos
https://api.github.com/users/es94129/starred{/owner}{/repo}
https://api.github.com/users/es94129/subscriptions
User
https://api.github.com/users/es94129
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5807). All of your documentation changes will be reflected on that endpoint.
2023-05-05T12:12:43Z
https://github.com/huggingface/datasets/pull/5807#issuecomment-1536168882
https://api.github.com/repos/huggingface/datasets/issues/5807
IC_kwDODunzps5bkBOy
https://api.github.com/repos/huggingface/datasets/issues/comments/1536168882/reactions
2023-05-05T12:12:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1536168882
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
Per the discussion in #5798, will implement with `joblibspark` instead.
2023-05-25T16:54:13Z
https://github.com/huggingface/datasets/pull/5807#issuecomment-1563221665
https://api.github.com/repos/huggingface/datasets/issues/5807
IC_kwDODunzps5dLN6h
https://api.github.com/repos/huggingface/datasets/issues/comments/1563221665/reactions
2023-05-25T16:54:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1563221665
https://avatars.githubusercontent.com/u/12763339?v=4
https://api.github.com/users/es94129/events{/privacy}
https://api.github.com/users/es94129/followers
https://api.github.com/users/es94129/following{/other_user}
https://api.github.com/users/es94129/gists{/gist_id}
https://github.com/es94129
es94129
MDQ6VXNlcjEyNzYzMzM5
https://api.github.com/users/es94129/orgs
https://api.github.com/users/es94129/received_events
https://api.github.com/users/es94129/repos
https://api.github.com/users/es94129/starred{/owner}{/repo}
https://api.github.com/users/es94129/subscriptions
User
https://api.github.com/users/es94129
public
As proposed in https://github.com/huggingface/datasets/issues/5798, this adds support to parallelized downloading in `load_dataset` with Spark, which can speed up the process by distributing the workload to worker nodes.

Parallelizing dataset processing is not supported in this PR.