Streaming dataset + interleave + DataLoader hangs with multiple workers
NONE
Same thing occurs when streaming files loaded from disk.
2022-03-23T14:36:32Z
https://github.com/huggingface/datasets/issues/3993#issuecomment-1076450712
https://api.github.com/repos/huggingface/datasets/issues/3993
IC_kwDODunzps5AKVWY
https://api.github.com/repos/huggingface/datasets/issues/comments/1076450712/reactions
2022-03-23T14:36:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1076450712
https://avatars.githubusercontent.com/u/614861?v=4
https://api.github.com/users/jpilaul/events{/privacy}
https://api.github.com/users/jpilaul/followers
https://api.github.com/users/jpilaul/following{/other_user}
https://api.github.com/users/jpilaul/gists{/gist_id}
https://github.com/jpilaul
jpilaul
MDQ6VXNlcjYxNDg2MQ==
https://api.github.com/users/jpilaul/orgs
https://api.github.com/users/jpilaul/received_events
https://api.github.com/users/jpilaul/repos
https://api.github.com/users/jpilaul/starred{/owner}{/repo}
https://api.github.com/users/jpilaul/subscriptions
User
https://api.github.com/users/jpilaul
public
MEMBER
Hi ! Thanks for reporting, could this be related to https://github.com/huggingface/datasets/issues/3950 ?

Currently streaming datasets only works in single process, but we're working on having in work in distributed setups as well :) (EDIT: done)
2022-03-28T12:48:21Z
https://github.com/huggingface/datasets/issues/3993#issuecomment-1080607909
https://api.github.com/repos/huggingface/datasets/issues/3993
IC_kwDODunzps5AaMSl
https://api.github.com/repos/huggingface/datasets/issues/comments/1080607909/reactions
2023-02-28T14:14:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1080607909
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, thanks for your reply. It seems related :)
2022-03-29T14:03:05Z
https://github.com/huggingface/datasets/issues/3993#issuecomment-1081912419
https://api.github.com/repos/huggingface/datasets/issues/3993
IC_kwDODunzps5AfKxj
https://api.github.com/repos/huggingface/datasets/issues/comments/1081912419/reactions
2022-03-29T14:03:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1081912419
https://avatars.githubusercontent.com/u/614861?v=4
https://api.github.com/users/jpilaul/events{/privacy}
https://api.github.com/users/jpilaul/followers
https://api.github.com/users/jpilaul/following{/other_user}
https://api.github.com/users/jpilaul/gists{/gist_id}
https://github.com/jpilaul
jpilaul
MDQ6VXNlcjYxNDg2MQ==
https://api.github.com/users/jpilaul/orgs
https://api.github.com/users/jpilaul/received_events
https://api.github.com/users/jpilaul/repos
https://api.github.com/users/jpilaul/starred{/owner}{/repo}
https://api.github.com/users/jpilaul/subscriptions
User
https://api.github.com/users/jpilaul
public
NONE
+1
2023-02-28T13:52:23Z
https://github.com/huggingface/datasets/issues/3993#issuecomment-1448215643
https://api.github.com/repos/huggingface/datasets/issues/3993
IC_kwDODunzps5WUgRb
https://api.github.com/repos/huggingface/datasets/issues/comments/1448215643/reactions
2023-02-28T13:52:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1448215643
https://avatars.githubusercontent.com/u/59222637?v=4
https://api.github.com/users/Mohammed20201991/events{/privacy}
https://api.github.com/users/Mohammed20201991/followers
https://api.github.com/users/Mohammed20201991/following{/other_user}
https://api.github.com/users/Mohammed20201991/gists{/gist_id}
https://github.com/Mohammed20201991
Mohammed20201991
MDQ6VXNlcjU5MjIyNjM3
https://api.github.com/users/Mohammed20201991/orgs
https://api.github.com/users/Mohammed20201991/received_events
https://api.github.com/users/Mohammed20201991/repos
https://api.github.com/users/Mohammed20201991/starred{/owner}{/repo}
https://api.github.com/users/Mohammed20201991/subscriptions
User
https://api.github.com/users/Mohammed20201991
public
MEMBER
Please update `datasets` if you're having this issue. What version are you using ?
2023-02-28T14:14:16Z
https://github.com/huggingface/datasets/issues/3993#issuecomment-1448252815
https://api.github.com/repos/huggingface/datasets/issues/3993
IC_kwDODunzps5WUpWP
https://api.github.com/repos/huggingface/datasets/issues/comments/1448252815/reactions
2023-02-28T14:14:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1448252815
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug
Interleaving multiple iterable datasets that use `load_dataset` on streaming mode hangs when passed to `torch.utils.data.DataLoader` with multiple workers.

## Steps to reproduce the bug
```python
from datasets import interleave_datasets, load_dataset
from torch.utils.data import DataLoader

en_dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)
fr_dataset = load_dataset('oscar', "unshuffled_deduplicated_fr", split='train', streaming=True)
it_dataset = load_dataset('oscar', "unshuffled_deduplicated_it", split='train', streaming=True)
de_dataset = load_dataset('oscar', "unshuffled_deduplicated_de", split='train', streaming=True)
multilingual_dataset = interleave_datasets([en_dataset, fr_dataset, de_dataset, it_dataset])
multilingual_dataset = multilingual_dataset.with_format('torch')

next(iter(multilingual_dataset))  # works fairly fast

dataloader = DataLoader(multilingual_dataset, batch_size=8, num_workers=4)
for batch in dataloader:
    print(len(batch))  # prints nothing after 30 min of waiting

dataloader = DataLoader(multilingual_dataset, batch_size=8, num_workers=0)
for batch in dataloader:
    print(len(batch))  # prints right away

```

## Expected results
It should be able to iterate the dataset with multiple workers.

## Actual results
Prints with results with `next(iter(multilingual_dataset)) ` and `num_workers=0`  but it prints nothing with `num_workers=4` or any number above 0.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.1.dev0
- `pytorch` version: 1.10.0+cu113
- Python version: 3.7
- PyArrow version: 6.0.1
