Add IterableDataset.shard()
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7252). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2024-10-25T11:09:41Z
https://github.com/huggingface/datasets/pull/7252#issuecomment-2437509938
https://api.github.com/repos/huggingface/datasets/issues/7252
IC_kwDODunzps6RSXMy
https://api.github.com/repos/huggingface/datasets/issues/comments/2437509938/reactions
2024-10-25T11:09:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2437509938
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
Is there some way to get this to work for pytorch dataloader workers?

eg. start with a single sharded IterableDataset.from_generator(), then reshard before calling map() to do expensive processing over multiple workers
2025-03-21T03:58:41Z
https://github.com/huggingface/datasets/pull/7252#issuecomment-2742189719
https://api.github.com/repos/huggingface/datasets/issues/7252
IC_kwDODunzps6jcn6X
https://api.github.com/repos/huggingface/datasets/issues/comments/2742189719/reactions
2025-03-21T03:58:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2742189719
https://avatars.githubusercontent.com/u/511073?v=4
https://api.github.com/users/jonathanasdf/events{/privacy}
https://api.github.com/users/jonathanasdf/followers
https://api.github.com/users/jonathanasdf/following{/other_user}
https://api.github.com/users/jonathanasdf/gists{/gist_id}
https://github.com/jonathanasdf
jonathanasdf
MDQ6VXNlcjUxMTA3Mw==
https://api.github.com/users/jonathanasdf/orgs
https://api.github.com/users/jonathanasdf/received_events
https://api.github.com/users/jonathanasdf/repos
https://api.github.com/users/jonathanasdf/starred{/owner}{/repo}
https://api.github.com/users/jonathanasdf/subscriptions
User
https://api.github.com/users/jonathanasdf
public
Will be useful to distribute a dataset across workers (other than pytorch) like spark

I also renamed `.n_shards` -> `.num_shards` for consistency and kept the old name for backward compatibility. And a few changes in internal functions for consistency as well (rank, world_size -> num_shards, index)

Breaking change: the new default for `contiguous` in `Dataset.shard()` is `True`, but imo not a big deal since I couldn't find any usage of `contiguous=False` internally (we always do contiguous=True for map-style datasets since its more optimized) or in the wild