IterableDataset: split by node and map may preprocess samples that will be skipped anyway
NONE
Does "number of shards" refer to the total number of data?

my config:
nproc_per_node=2
ds=ds['train'] = load_dataset(streaming=True).take(50000)

I'm test again: in prepare_data(),  data have the same for each GPU

2023-06-15T10:50:41Z
https://github.com/huggingface/datasets/issues/5961#issuecomment-1592811547
https://api.github.com/repos/huggingface/datasets/issues/5961
IC_kwDODunzps5e8GAb
https://api.github.com/repos/huggingface/datasets/issues/comments/1592811547/reactions
2023-06-15T10:50:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1592811547
https://avatars.githubusercontent.com/u/27708347?v=4
https://api.github.com/users/johnchienbronci/events{/privacy}
https://api.github.com/users/johnchienbronci/followers
https://api.github.com/users/johnchienbronci/following{/other_user}
https://api.github.com/users/johnchienbronci/gists{/gist_id}
https://github.com/johnchienbronci
johnchienbronci
MDQ6VXNlcjI3NzA4MzQ3
https://api.github.com/users/johnchienbronci/orgs
https://api.github.com/users/johnchienbronci/received_events
https://api.github.com/users/johnchienbronci/repos
https://api.github.com/users/johnchienbronci/starred{/owner}{/repo}
https://api.github.com/users/johnchienbronci/subscriptions
User
https://api.github.com/users/johnchienbronci
public
MEMBER
The number of shards is `ds.n_shards`. It corresponds generally to the number of files the dataset is made of, to be able to distribute to several nodes.

**You don't end up with the same data per GPU**. But all the samples are going through your preprocessing function you pass to map. They are just skipped afterwards to only keep 1 sample out of n(GPUs)
2023-06-15T11:00:31Z
https://github.com/huggingface/datasets/issues/5961#issuecomment-1592831128
https://api.github.com/repos/huggingface/datasets/issues/5961
IC_kwDODunzps5e8KyY
https://api.github.com/repos/huggingface/datasets/issues/comments/1592831128/reactions
2023-06-15T11:00:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1592831128
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end. 
Is my understanding correct?

Where can I print the actual training data for each GPU?
2023-06-15T11:17:36Z
https://github.com/huggingface/datasets/issues/5961#issuecomment-1592854470
https://api.github.com/repos/huggingface/datasets/issues/5961
IC_kwDODunzps5e8QfG
https://api.github.com/repos/huggingface/datasets/issues/comments/1592854470/reactions
2023-06-15T11:17:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1592854470
https://avatars.githubusercontent.com/u/27708347?v=4
https://api.github.com/users/johnchienbronci/events{/privacy}
https://api.github.com/users/johnchienbronci/followers
https://api.github.com/users/johnchienbronci/following{/other_user}
https://api.github.com/users/johnchienbronci/gists{/gist_id}
https://github.com/johnchienbronci
johnchienbronci
MDQ6VXNlcjI3NzA4MzQ3
https://api.github.com/users/johnchienbronci/orgs
https://api.github.com/users/johnchienbronci/received_events
https://api.github.com/users/johnchienbronci/repos
https://api.github.com/users/johnchienbronci/starred{/owner}{/repo}
https://api.github.com/users/johnchienbronci/subscriptions
User
https://api.github.com/users/johnchienbronci
public
MEMBER
> For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
Is my understanding correct?

Yes exactly :)

> Where can I print the actual training data for each GPU?

You should call print in the data_collator
2023-06-15T11:39:25Z
https://github.com/huggingface/datasets/issues/5961#issuecomment-1592880136
https://api.github.com/repos/huggingface/datasets/issues/5961
IC_kwDODunzps5e8WwI
https://api.github.com/repos/huggingface/datasets/issues/comments/1592880136/reactions
2023-06-15T11:39:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1592880136
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I print out n_shards, and under multiple GPUs, this value is always 1.
Is this value correct?
2023-06-17T08:45:06Z
https://github.com/huggingface/datasets/issues/5961#issuecomment-1595674774
https://api.github.com/repos/huggingface/datasets/issues/5961
IC_kwDODunzps5fHBCW
https://api.github.com/repos/huggingface/datasets/issues/comments/1595674774/reactions
2023-06-17T08:45:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1595674774
https://avatars.githubusercontent.com/u/27708347?v=4
https://api.github.com/users/johnchienbronci/events{/privacy}
https://api.github.com/users/johnchienbronci/followers
https://api.github.com/users/johnchienbronci/following{/other_user}
https://api.github.com/users/johnchienbronci/gists{/gist_id}
https://github.com/johnchienbronci
johnchienbronci
MDQ6VXNlcjI3NzA4MzQ3
https://api.github.com/users/johnchienbronci/orgs
https://api.github.com/users/johnchienbronci/received_events
https://api.github.com/users/johnchienbronci/repos
https://api.github.com/users/johnchienbronci/starred{/owner}{/repo}
https://api.github.com/users/johnchienbronci/subscriptions
User
https://api.github.com/users/johnchienbronci
public
MEMBER
Yes it's correct, and it explains why you always have the same data passed to your map function (the data can't be split).

But after being passed to `map`, each GPU keeps one example out of n(GPUs) so that you don't end up with duplicate data across GPUs
2023-06-19T11:08:52Z
https://github.com/huggingface/datasets/issues/5961#issuecomment-1596987676
https://api.github.com/repos/huggingface/datasets/issues/5961
IC_kwDODunzps5fMBkc
https://api.github.com/repos/huggingface/datasets/issues/comments/1596987676/reactions
2023-06-19T11:08:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1596987676
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> > For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
> > Is my understanding correct?
> 
> Yes exactly :)
> 
> > Where can I print the actual training data for each GPU?
> 
> You should call print in the data_collator

OK, when printing the train data in the data collator, each GPU sees different data.

Thanks for your reply
2023-06-20T01:30:40Z
https://github.com/huggingface/datasets/issues/5961#issuecomment-1597994829
https://api.github.com/repos/huggingface/datasets/issues/5961
IC_kwDODunzps5fP3dN
https://api.github.com/repos/huggingface/datasets/issues/comments/1597994829/reactions
2023-06-20T01:30:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1597994829
https://avatars.githubusercontent.com/u/27708347?v=4
https://api.github.com/users/johnchienbronci/events{/privacy}
https://api.github.com/users/johnchienbronci/followers
https://api.github.com/users/johnchienbronci/following{/other_user}
https://api.github.com/users/johnchienbronci/gists{/gist_id}
https://github.com/johnchienbronci
johnchienbronci
MDQ6VXNlcjI3NzA4MzQ3
https://api.github.com/users/johnchienbronci/orgs
https://api.github.com/users/johnchienbronci/received_events
https://api.github.com/users/johnchienbronci/repos
https://api.github.com/users/johnchienbronci/starred{/owner}{/repo}
https://api.github.com/users/johnchienbronci/subscriptions
User
https://api.github.com/users/johnchienbronci
public
NONE
Do we have a solution for this one? Or it's required to get "number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU"
2023-09-01T01:00:40Z
https://github.com/huggingface/datasets/issues/5961#issuecomment-1701967380
https://api.github.com/repos/huggingface/datasets/issues/5961
IC_kwDODunzps5lcfYU
https://api.github.com/repos/huggingface/datasets/issues/comments/1701967380/reactions
2023-09-01T01:00:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1701967380
https://avatars.githubusercontent.com/u/6926994?v=4
https://api.github.com/users/chenxingyu-cs/events{/privacy}
https://api.github.com/users/chenxingyu-cs/followers
https://api.github.com/users/chenxingyu-cs/following{/other_user}
https://api.github.com/users/chenxingyu-cs/gists{/gist_id}
https://github.com/chenxingyu-cs
chenxingyu-cs
MDQ6VXNlcjY5MjY5OTQ=
https://api.github.com/users/chenxingyu-cs/orgs
https://api.github.com/users/chenxingyu-cs/received_events
https://api.github.com/users/chenxingyu-cs/repos
https://api.github.com/users/chenxingyu-cs/starred{/owner}{/repo}
https://api.github.com/users/chenxingyu-cs/subscriptions
User
https://api.github.com/users/chenxingyu-cs
public
MEMBER
For now it's required to have a number of shards that is a factor of the number of GPUs to not have all the workers process the same data (and then skip the right ones to not end up training on duplicate data).

It would be quite complex to implement a strategy that would utilize all the GPUs with an arbitrary number of shards even at the end of training
2023-09-01T10:35:11Z
https://github.com/huggingface/datasets/issues/5961#issuecomment-1702538136
https://api.github.com/repos/huggingface/datasets/issues/5961
IC_kwDODunzps5lequY
https://api.github.com/repos/huggingface/datasets/issues/comments/1702538136/reactions
2023-09-01T10:35:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1702538136
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
              There are two ways an iterable dataset can be split by node:
1. if the number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU
2. otherwise, each GPU iterate on the data and at the end keeps 1 sample out of n(GPUs) - skipping the others.

In case 2. it's therefore possible to have the same examples passed to `prepare_dataset` for each GPU.

This doesn't sound optimized though, because it runs the preprocessing on samples that won't be used in the end.

Could you open a new issue so that we can discuss about this and find a solution ?

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/5360#issuecomment-1592729051_
            