Distributed data parallel training for streaming datasets
MEMBER
Hi ! According to https://huggingface.co/docs/datasets/use_with_pytorch#stream-data you can use the pytorch DataLoader with `num_workers>0` to distribute the shards across your workers (it uses `torch.utils.data.get_worker_info()` to get the worker ID and select the right subsets of shards to use)

<s> EDIT: here is a code example </s>
```python
# ds = ds.with_format("torch")
# dataloader = DataLoader(ds, num_workers=num_workers)
```

EDIT: `with_format("torch")` is not required, now you can just do
```python
dataloader = DataLoader(ds, num_workers=num_workers)
```
2022-07-25T16:51:30Z
https://github.com/huggingface/datasets/issues/4694#issuecomment-1194344816
https://api.github.com/repos/huggingface/datasets/issues/4694
IC_kwDODunzps5HMEFw
https://api.github.com/repos/huggingface/datasets/issues/comments/1194344816/reactions
2023-02-28T14:15:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1194344816
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@cyk1337 does streaming datasets with multi-gpu works for you? I am testing on one node with multiple gpus, but this is freezing, https://github.com/huggingface/datasets/issues/5123 
In case you could make this work, could you share with me your data-loading codes?
thank you
2022-10-24T16:18:40Z
https://github.com/huggingface/datasets/issues/4694#issuecomment-1289277716
https://api.github.com/repos/huggingface/datasets/issues/4694
IC_kwDODunzps5M2NEU
https://api.github.com/repos/huggingface/datasets/issues/comments/1289277716/reactions
2022-10-24T16:20:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1289277716
https://avatars.githubusercontent.com/u/59409879?v=4
https://api.github.com/users/jackfeinmann5/events{/privacy}
https://api.github.com/users/jackfeinmann5/followers
https://api.github.com/users/jackfeinmann5/following{/other_user}
https://api.github.com/users/jackfeinmann5/gists{/gist_id}
https://github.com/jackfeinmann5
jackfeinmann5
MDQ6VXNlcjU5NDA5ODc5
https://api.github.com/users/jackfeinmann5/orgs
https://api.github.com/users/jackfeinmann5/received_events
https://api.github.com/users/jackfeinmann5/repos
https://api.github.com/users/jackfeinmann5/starred{/owner}{/repo}
https://api.github.com/users/jackfeinmann5/subscriptions
User
https://api.github.com/users/jackfeinmann5
public
NONE
+1
2023-02-28T13:48:23Z
https://github.com/huggingface/datasets/issues/4694#issuecomment-1448209927
https://api.github.com/repos/huggingface/datasets/issues/4694
IC_kwDODunzps5WUe4H
https://api.github.com/repos/huggingface/datasets/issues/comments/1448209927/reactions
2023-02-28T13:48:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1448209927
https://avatars.githubusercontent.com/u/59222637?v=4
https://api.github.com/users/Mohammed20201991/events{/privacy}
https://api.github.com/users/Mohammed20201991/followers
https://api.github.com/users/Mohammed20201991/following{/other_user}
https://api.github.com/users/Mohammed20201991/gists{/gist_id}
https://github.com/Mohammed20201991
Mohammed20201991
MDQ6VXNlcjU5MjIyNjM3
https://api.github.com/users/Mohammed20201991/orgs
https://api.github.com/users/Mohammed20201991/received_events
https://api.github.com/users/Mohammed20201991/repos
https://api.github.com/users/Mohammed20201991/starred{/owner}{/repo}
https://api.github.com/users/Mohammed20201991/subscriptions
User
https://api.github.com/users/Mohammed20201991
public
MEMBER
This has been implemented in `datasets` 2.8:
```python
from datasets.distributed import split_dataset_by_node

ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
```

docs: https://huggingface.co/docs/datasets/use_with_pytorch#distributed
2023-02-28T14:16:50Z
https://github.com/huggingface/datasets/issues/4694#issuecomment-1448257255
https://api.github.com/repos/huggingface/datasets/issues/4694
IC_kwDODunzps5WUqbn
https://api.github.com/repos/huggingface/datasets/issues/comments/1448257255/reactions
2023-02-28T14:17:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1448257255
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
i'm having hanging issues with this when using DDP and allocating the datasets with `split_dataset_by_node` 🤔

--- 
### edit
I don't want to pollute this thread, but for the sake of following up, I observed hanging close to the final iteration of the dataloader. I think this was happening on the final shard. First, I removed the final shard and things worked. Then (including all shards), I reordered the list of shards: `load_dataset('json', data_files=reordered, streaming=True)` and no hang. 

I won't open an issue yet bc I am not quite sure about this observation.
2023-04-26T05:46:08Z
https://github.com/huggingface/datasets/issues/4694#issuecomment-1522819538
https://api.github.com/repos/huggingface/datasets/issues/4694
IC_kwDODunzps5axGHS
https://api.github.com/repos/huggingface/datasets/issues/comments/1522819538/reactions
2023-04-26T18:21:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1522819538
https://avatars.githubusercontent.com/u/28735574?v=4
https://api.github.com/users/wconnell/events{/privacy}
https://api.github.com/users/wconnell/followers
https://api.github.com/users/wconnell/following{/other_user}
https://api.github.com/users/wconnell/gists{/gist_id}
https://github.com/wconnell
wconnell
MDQ6VXNlcjI4NzM1NTc0
https://api.github.com/users/wconnell/orgs
https://api.github.com/users/wconnell/received_events
https://api.github.com/users/wconnell/repos
https://api.github.com/users/wconnell/starred{/owner}{/repo}
https://api.github.com/users/wconnell/subscriptions
User
https://api.github.com/users/wconnell
public
MEMBER
@wconnell would you mind opening a different bug issue and giving more details?
https://github.com/huggingface/datasets/issues/new?assignees=&labels=&template=bug-report.yml

Thanks.
2023-04-26T08:57:51Z
https://github.com/huggingface/datasets/issues/4694#issuecomment-1523037690
https://api.github.com/repos/huggingface/datasets/issues/4694
IC_kwDODunzps5ax7X6
https://api.github.com/repos/huggingface/datasets/issues/comments/1523037690/reactions
2023-04-26T08:57:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1523037690
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
### Feature request

Any documentations for the the `load_dataset(streaming=True)` for (multi-node multi-GPU) DDP training? 

### Motivation

Given a bunch of data files, it is expected to split them onto different GPUs. Is there a guide or documentation?

### Your contribution

Does it requires manually split on data files for each worker in `DatasetBuilder._split_generator()`? What is`IterableDatasetShard` expected to do?