How to implement DistributedSampler with datasets 
NONE
Hi Apparently I need to shard the data and give one host a chunk, could you provide me please with examples on how to do it? I want to use it jointly with finetune_trainer.py in huggingface repo seq2seq examples. thanks. 
2020-11-09T08:33:16Z
https://github.com/huggingface/datasets/issues/813#issuecomment-723854973
https://api.github.com/repos/huggingface/datasets/issues/813
MDEyOklzc3VlQ29tbWVudDcyMzg1NDk3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/723854973/reactions
2020-11-09T08:33:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/723854973
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
NONE
Hey @rabeehkarimimahabadi I'm actually looking for the same feature. Did you manage to get somewhere?
2021-01-20T13:42:27Z
https://github.com/huggingface/datasets/issues/813#issuecomment-763613374
https://api.github.com/repos/huggingface/datasets/issues/813
MDEyOklzc3VlQ29tbWVudDc2MzYxMzM3NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/763613374/reactions
2021-01-20T13:42:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/763613374
https://avatars.githubusercontent.com/u/32683010?v=4
https://api.github.com/users/JulesBelveze/events{/privacy}
https://api.github.com/users/JulesBelveze/followers
https://api.github.com/users/JulesBelveze/following{/other_user}
https://api.github.com/users/JulesBelveze/gists{/gist_id}
https://github.com/JulesBelveze
JulesBelveze
MDQ6VXNlcjMyNjgzMDEw
https://api.github.com/users/JulesBelveze/orgs
https://api.github.com/users/JulesBelveze/received_events
https://api.github.com/users/JulesBelveze/repos
https://api.github.com/users/JulesBelveze/starred{/owner}{/repo}
https://api.github.com/users/JulesBelveze/subscriptions
User
https://api.github.com/users/JulesBelveze
public
NONE
@rabeehkarimimahabadi need the same feature
2021-10-25T10:36:38Z
https://github.com/huggingface/datasets/issues/813#issuecomment-950780284
https://api.github.com/repos/huggingface/datasets/issues/813
IC_kwDODunzps44q8F8
https://api.github.com/repos/huggingface/datasets/issues/comments/950780284/reactions
2021-10-25T10:36:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/950780284
https://avatars.githubusercontent.com/u/15981416?v=4
https://api.github.com/users/vanpersie32/events{/privacy}
https://api.github.com/users/vanpersie32/followers
https://api.github.com/users/vanpersie32/following{/other_user}
https://api.github.com/users/vanpersie32/gists{/gist_id}
https://github.com/vanpersie32
vanpersie32
MDQ6VXNlcjE1OTgxNDE2
https://api.github.com/users/vanpersie32/orgs
https://api.github.com/users/vanpersie32/received_events
https://api.github.com/users/vanpersie32/repos
https://api.github.com/users/vanpersie32/starred{/owner}{/repo}
https://api.github.com/users/vanpersie32/subscriptions
User
https://api.github.com/users/vanpersie32
public
COLLABORATOR
Hi! I think you can use the `accelerate` library for that, which implements distributed sampling.
2022-10-05T12:54:18Z
https://github.com/huggingface/datasets/issues/813#issuecomment-1268397571
https://api.github.com/repos/huggingface/datasets/issues/813
IC_kwDODunzps5LmjYD
https://api.github.com/repos/huggingface/datasets/issues/comments/1268397571/reactions
2022-10-05T12:54:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1268397571
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
Hi,
I am using your datasets to define my dataloaders, and I am training finetune_trainer.py in huggingface repo on them.
I need a distributedSampler to be able to train the models on TPUs being able to distribute the load across the TPU cores. Could you tell me how I can implement the distribued sampler when using datasets in which datasets are iterative? To give you more context, I have multiple of datasets and I need to write sampler for this case. thanks. 