making sure datasets are not loaded in memory and distributed training of them
CONTRIBUTOR
my implementation of sharding per TPU core: https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/trainers/t5_trainer.py#L316 
my implementation of dataloader for this case https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/tasks/tasks.py#L496 
2020-12-02T08:50:34Z
https://github.com/huggingface/datasets/issues/988#issuecomment-737085941
https://api.github.com/repos/huggingface/datasets/issues/988
MDEyOklzc3VlQ29tbWVudDczNzA4NTk0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/737085941/reactions
2020-12-02T08:50:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/737085941
https://avatars.githubusercontent.com/u/6278280?v=4
https://api.github.com/users/rabeehk/events{/privacy}
https://api.github.com/users/rabeehk/followers
https://api.github.com/users/rabeehk/following{/other_user}
https://api.github.com/users/rabeehk/gists{/gist_id}
https://github.com/rabeehk
rabeehk
MDQ6VXNlcjYyNzgyODA=
https://api.github.com/users/rabeehk/orgs
https://api.github.com/users/rabeehk/received_events
https://api.github.com/users/rabeehk/repos
https://api.github.com/users/rabeehk/starred{/owner}{/repo}
https://api.github.com/users/rabeehk/subscriptions
User
https://api.github.com/users/rabeehk
public
COLLABORATOR
Hi!  You can use the `assert not bool(dataset.cache_files)` assertion to ensure your data is in memory. And I suggest using `accelerate` for distributed training.
2022-10-05T13:00:42Z
https://github.com/huggingface/datasets/issues/988#issuecomment-1268404715
https://api.github.com/repos/huggingface/datasets/issues/988
IC_kwDODunzps5LmlHr
https://api.github.com/repos/huggingface/datasets/issues/comments/1268404715/reactions
2022-10-05T13:00:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1268404715
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
Hi
I am dealing with large-scale datasets which I need to train distributedly, I used the shard function to divide the dataset across the cores, without any sampler, this does not work for distributed training and does not become any faster than 1 TPU core. 1) how I can make sure data is not loaded in memory 2) in case of distributed training with iterative datasets which measures needs to be taken? Is this all sharding the data only. I was wondering if there can be possibility for me to discuss this with someone with distributed training with iterative datasets using dataset library. thanks 