Why not use self._epoch as seed to shuffle in distributed training with IterableDataset
CONTRIBUTOR
@lhoestq 
2023-08-17T11:09:50Z
https://github.com/huggingface/datasets/issues/6156#issuecomment-1682090984
https://api.github.com/repos/huggingface/datasets/issues/6156
IC_kwDODunzps5kQqvo
https://api.github.com/repos/huggingface/datasets/issues/comments/1682090984/reactions
2023-08-17T11:09:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1682090984
https://avatars.githubusercontent.com/u/11533479?v=4
https://api.github.com/users/npuichigo/events{/privacy}
https://api.github.com/users/npuichigo/followers
https://api.github.com/users/npuichigo/following{/other_user}
https://api.github.com/users/npuichigo/gists{/gist_id}
https://github.com/npuichigo
npuichigo
MDQ6VXNlcjExNTMzNDc5
https://api.github.com/users/npuichigo/orgs
https://api.github.com/users/npuichigo/received_events
https://api.github.com/users/npuichigo/repos
https://api.github.com/users/npuichigo/starred{/owner}{/repo}
https://api.github.com/users/npuichigo/subscriptions
User
https://api.github.com/users/npuichigo
public
MEMBER
`_effective_generator` returns a RNG that takes into account `self._epoch` and the current dataset's base shuffling RNG (which can be set by specifying `seed=` in `.shuffle() for example`).

To fix your error you can pass `seed=` to `.shuffle()`. And the shuffling will depend on both this seed and `self._epoch`
2023-08-17T12:37:59Z
https://github.com/huggingface/datasets/issues/6156#issuecomment-1682212264
https://api.github.com/repos/huggingface/datasets/issues/6156
IC_kwDODunzps5kRIWo
https://api.github.com/repos/huggingface/datasets/issues/comments/1682212264/reactions
2023-08-17T12:40:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1682212264
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks for the reply
2023-08-17T14:33:14Z
https://github.com/huggingface/datasets/issues/6156#issuecomment-1682396715
https://api.github.com/repos/huggingface/datasets/issues/6156
IC_kwDODunzps5kR1Yr
https://api.github.com/repos/huggingface/datasets/issues/comments/1682396715/reactions
2023-08-17T14:33:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1682396715
https://avatars.githubusercontent.com/u/11533479?v=4
https://api.github.com/users/npuichigo/events{/privacy}
https://api.github.com/users/npuichigo/followers
https://api.github.com/users/npuichigo/following{/other_user}
https://api.github.com/users/npuichigo/gists{/gist_id}
https://github.com/npuichigo
npuichigo
MDQ6VXNlcjExNTMzNDc5
https://api.github.com/users/npuichigo/orgs
https://api.github.com/users/npuichigo/received_events
https://api.github.com/users/npuichigo/repos
https://api.github.com/users/npuichigo/starred{/owner}{/repo}
https://api.github.com/users/npuichigo/subscriptions
User
https://api.github.com/users/npuichigo
public
### Describe the bug

Currently, distributed training with `IterableDataset` needs to pass fixed seed to shuffle to keep each node use the same seed to avoid overlapping.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1174-L1177

My question is why not directly use `self._epoch` which is set by `set_epoch` as seed? It's almost the same across nodes.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1790-L1801

If not using `self._epoch` as shuffling seed, what does this method do to prepare an epoch seeded generator?
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1206



### Steps to reproduce the bug

As mentioned above.

### Expected behavior

As mentioned above.

### Environment info

Not related