Make IterableDataset (optionally) resumable
CONTRIBUTOR
@lhoestq Hi again~ Just circling back on this
Wondering if there’s anything I can do to help move this forward. 🤗 
Thanks!
2025-02-06T07:40:17Z
https://github.com/huggingface/datasets/pull/7385#issuecomment-2639052751
https://api.github.com/repos/huggingface/datasets/issues/7385
IC_kwDODunzps6dTL_P
https://api.github.com/repos/huggingface/datasets/issues/comments/2639052751/reactions
2025-02-06T07:40:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2639052751
https://avatars.githubusercontent.com/u/18402347?v=4
https://api.github.com/users/yzhangcs/events{/privacy}
https://api.github.com/users/yzhangcs/followers
https://api.github.com/users/yzhangcs/following{/other_user}
https://api.github.com/users/yzhangcs/gists{/gist_id}
https://github.com/yzhangcs
yzhangcs
MDQ6VXNlcjE4NDAyMzQ3
https://api.github.com/users/yzhangcs/orgs
https://api.github.com/users/yzhangcs/received_events
https://api.github.com/users/yzhangcs/repos
https://api.github.com/users/yzhangcs/starred{/owner}{/repo}
https://api.github.com/users/yzhangcs/subscriptions
User
https://api.github.com/users/yzhangcs
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7385). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2025-03-03T17:31:39Z
https://github.com/huggingface/datasets/pull/7385#issuecomment-2695093614
https://api.github.com/repos/huggingface/datasets/issues/7385
IC_kwDODunzps6go91u
https://api.github.com/repos/huggingface/datasets/issues/comments/2695093614/reactions
2025-03-03T17:31:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2695093614
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
### What does this PR do?
This PR introduces a new `stateful` option to the `dataset.shuffle` method, which defaults to `False`. 
When enabled, this option allows for resumable shuffling of `IterableDataset` instances, albeit with some additional memory overhead.

Key points:
* All tests have passed
* Docstrings have been updated to reflect the new functionality

I'm very looking forward to receiving feedback on this implementation! @lhoestq