[load_dataset] shard and parallelize the process
NONE
I need the same feature for distributed training
2021-10-25T10:39:42Z
https://github.com/huggingface/datasets/issues/2650#issuecomment-950782740
https://api.github.com/repos/huggingface/datasets/issues/2650
IC_kwDODunzps44q8sU
https://api.github.com/repos/huggingface/datasets/issues/comments/950782740/reactions
2021-10-25T10:39:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/950782740
https://avatars.githubusercontent.com/u/15981416?v=4
https://api.github.com/users/vanpersie32/events{/privacy}
https://api.github.com/users/vanpersie32/followers
https://api.github.com/users/vanpersie32/following{/other_user}
https://api.github.com/users/vanpersie32/gists{/gist_id}
https://github.com/vanpersie32
vanpersie32
MDQ6VXNlcjE1OTgxNDE2
https://api.github.com/users/vanpersie32/orgs
https://api.github.com/users/vanpersie32/received_events
https://api.github.com/users/vanpersie32/repos
https://api.github.com/users/vanpersie32/starred{/owner}{/repo}
https://api.github.com/users/vanpersie32/subscriptions
User
https://api.github.com/users/vanpersie32
public
MEMBER
I think @TevenLeScao is exploring adding multiprocessing in `GeneratorBasedBuilder._prepare_split` - feel free to post updates here :)
2022-10-07T16:10:43Z
https://github.com/huggingface/datasets/issues/2650#issuecomment-1271786241
https://api.github.com/repos/huggingface/datasets/issues/2650
IC_kwDODunzps5LzesB
https://api.github.com/repos/huggingface/datasets/issues/comments/1271786241/reactions
2022-10-07T16:10:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1271786241
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Posted a PR to address the building side, still needs something to load sharded arrow files + tests
2022-10-12T20:05:07Z
https://github.com/huggingface/datasets/issues/2650#issuecomment-1276675015
https://api.github.com/repos/huggingface/datasets/issues/2650
IC_kwDODunzps5MGIPH
https://api.github.com/repos/huggingface/datasets/issues/comments/1276675015/reactions
2022-10-12T20:05:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1276675015
https://avatars.githubusercontent.com/u/26709476?v=4
https://api.github.com/users/TevenLeScao/events{/privacy}
https://api.github.com/users/TevenLeScao/followers
https://api.github.com/users/TevenLeScao/following{/other_user}
https://api.github.com/users/TevenLeScao/gists{/gist_id}
https://github.com/TevenLeScao
TevenLeScao
MDQ6VXNlcjI2NzA5NDc2
https://api.github.com/users/TevenLeScao/orgs
https://api.github.com/users/TevenLeScao/received_events
https://api.github.com/users/TevenLeScao/repos
https://api.github.com/users/TevenLeScao/starred{/owner}{/repo}
https://api.github.com/users/TevenLeScao/subscriptions
User
https://api.github.com/users/TevenLeScao
public
COLLABORATOR
Closing as this feature has been implemented in #5107
2023-11-28T19:11:41Z
https://github.com/huggingface/datasets/issues/2650#issuecomment-1830512720
https://api.github.com/repos/huggingface/datasets/issues/2650
IC_kwDODunzps5tG2hQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1830512720/reactions
2023-11-28T19:11:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1830512720
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
- Some huge datasets take forever to build the first time. (e.g. oscar/en) as it's done in a single cpu core.
- If the build crashes, everything done up to that point gets lost

Request: Shard the build over multiple arrow files, which would enable:
- much faster build by parallelizing the build process
- if the process crashed, the completed arrow files don't need to be re-built again

Thank you!

@lhoestq 