shuffle with torch generator 
NONE
Is there a way one can convert the two generator? not sure overall what alternatives I could have to shuffle the datasets with a torch generator, thanks 
2020-12-21T08:34:52Z
https://github.com/huggingface/datasets/issues/1611#issuecomment-748845500
https://api.github.com/repos/huggingface/datasets/issues/1611
MDEyOklzc3VlQ29tbWVudDc0ODg0NTUwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/748845500/reactions
2020-12-21T08:34:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/748845500
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
NONE
@lhoestq  let me please expalin in more details, maybe you could help me suggesting an alternative to solve the issue for now, I have multiple large datasets using huggingface library, then I need to define a distributed sampler on top of it, for this I need to shard the datasets and give each shard to each core, but before sharding I need to shuffle the dataset, if you are familiar with distributed sampler in pytorch, this needs to be done based on seed+epoch generator to make it consistent across the cores they do it through defining a torch generator, I was wondering if you could tell me how I can shuffle the data for now, I am unfortunately blocked by this and have a limited time left, and I greatly appreciate your help on this. thanks 
2020-12-21T15:35:49Z
https://github.com/huggingface/datasets/issues/1611#issuecomment-749036197
https://api.github.com/repos/huggingface/datasets/issues/1611
MDEyOklzc3VlQ29tbWVudDc0OTAzNjE5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/749036197/reactions
2020-12-21T15:42:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749036197
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
NONE
@lhoestq  Is there a way I could shuffle the datasets from this library with a custom defined shuffle function? thanks for your help on this. 
2020-12-21T15:39:31Z
https://github.com/huggingface/datasets/issues/1611#issuecomment-749037921
https://api.github.com/repos/huggingface/datasets/issues/1611
MDEyOklzc3VlQ29tbWVudDc0OTAzNzkyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/749037921/reactions
2020-12-21T15:39:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749037921
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
MEMBER
Right now the shuffle method only accepts the `seed` (optional int) or `generator` (optional `np.random.Generator`) parameters.

Here is a suggestion to shuffle the data using your own shuffle method using `select`.
`select` can be used to re-order the dataset samples or simply pick a few ones if you want.
It's what is used under the hood when you call `dataset.shuffle`.

To use `select` you must have the list of re-ordered indices of your samples.

Let's say you have a `shuffle` methods that you want to use. Then you can first build your shuffled list of indices:
```python
shuffled_indices = shuffle(range(len(dataset)))
```

Then you can shuffle your dataset using the shuffled indices with 
```python
shuffled_dataset = dataset.select(shuffled_indices)
```

Hope that helps
2020-12-21T15:45:21Z
https://github.com/huggingface/datasets/issues/1611#issuecomment-749040652
https://api.github.com/repos/huggingface/datasets/issues/1611
MDEyOklzc3VlQ29tbWVudDc0OTA0MDY1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/749040652/reactions
2020-12-21T15:45:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749040652
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
thank you @lhoestq  thank you very much for responding to my question, this greatly helped me and remove the blocking for continuing my work, thanks. 
2020-12-21T15:52:42Z
https://github.com/huggingface/datasets/issues/1611#issuecomment-749044535
https://api.github.com/repos/huggingface/datasets/issues/1611
MDEyOklzc3VlQ29tbWVudDc0OTA0NDUzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/749044535/reactions
2020-12-21T15:52:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749044535
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
NONE
@lhoestq  could you confirm the method proposed does not bring the whole data into memory? thanks 
2020-12-21T17:26:21Z
https://github.com/huggingface/datasets/issues/1611#issuecomment-749097858
https://api.github.com/repos/huggingface/datasets/issues/1611
MDEyOklzc3VlQ29tbWVudDc0OTA5Nzg1OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/749097858/reactions
2020-12-21T17:26:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749097858
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
MEMBER
Yes the dataset is not loaded into memory
2020-12-21T17:28:40Z
https://github.com/huggingface/datasets/issues/1611#issuecomment-749098982
https://api.github.com/repos/huggingface/datasets/issues/1611
MDEyOklzc3VlQ29tbWVudDc0OTA5ODk4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/749098982/reactions
2020-12-21T17:28:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749098982
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
great. thanks a lot.
2020-12-21T18:12:19Z
https://github.com/huggingface/datasets/issues/1611#issuecomment-749118827
https://api.github.com/repos/huggingface/datasets/issues/1611
MDEyOklzc3VlQ29tbWVudDc0OTExODgyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/749118827/reactions
2020-12-21T18:12:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749118827
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
Hi
I need to shuffle mutliple large datasets with `generator = torch.Generator()` for a distributed sampler which needs to make sure datasets are consistent across different cores, for this, this is really necessary for me to use  torch generator, based on documentation this generator is not supported with datasets, I really need to make shuffle work with this generator and I was wondering what I can do about this issue, thanks for your help 

@lhoestq 