data duplicate when setting num_works > 1 with streaming data
MEMBER
Hi ! Thanks for reporting :)

When using a PyTorch's data loader with `num_workers>1` and an iterable dataset, each worker streams the exact same data by default, resulting in duplicate data when iterating using the data loader.

We can probably fix this in `datasets` by checking `torch.utils.data.get_worker_info()` which gives the worker id if it happens.
2021-12-13T18:56:44Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-992773068
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps47LIPM
https://api.github.com/repos/huggingface/datasets/issues/comments/992773068/reactions
2021-12-13T18:56:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/992773068
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Hi ! Thanks for reporting :)
> 
> When using a PyTorch's data loader with `num_workers>1` and an iterable dataset, each worker streams the exact same data by default, resulting in duplicate data when iterating using the data loader.
> 
> We can probably fix this in `datasets` by checking `torch.utils.data.get_worker_info()` which gives the worker id if it happens.
Hi ! Thanks for reply

Do u have some plans to fix the problem?

2021-12-14T06:29:50Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-993202239
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps47MxA_
https://api.github.com/repos/huggingface/datasets/issues/comments/993202239/reactions
2021-12-14T06:29:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/993202239
https://avatars.githubusercontent.com/u/16486492?v=4
https://api.github.com/users/cloudyuyuyu/events{/privacy}
https://api.github.com/users/cloudyuyuyu/followers
https://api.github.com/users/cloudyuyuyu/following{/other_user}
https://api.github.com/users/cloudyuyuyu/gists{/gist_id}
https://github.com/cloudyuyuyu
cloudyuyuyu
MDQ6VXNlcjE2NDg2NDky
https://api.github.com/users/cloudyuyuyu/orgs
https://api.github.com/users/cloudyuyuyu/received_events
https://api.github.com/users/cloudyuyuyu/repos
https://api.github.com/users/cloudyuyuyu/starred{/owner}{/repo}
https://api.github.com/users/cloudyuyuyu/subscriptions
User
https://api.github.com/users/cloudyuyuyu
public
MEMBER
Isn’t that somehow a bug on PyTorch side? (Just asking because this behavior seems quite general and maybe not what would be intended)
2021-12-14T09:13:13Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-993330598
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps47NQWm
https://api.github.com/repos/huggingface/datasets/issues/comments/993330598/reactions
2021-12-14T09:13:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/993330598
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
From PyTorch's documentation [here](https://pytorch.org/docs/stable/data.html#dataset-types):

> When using an IterableDataset with multi-process data loading. The same dataset object is replicated on each worker process, and thus the replicas must be configured differently to avoid duplicated data. See [IterableDataset](https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset) documentations for how to achieve this.

It looks like an intended behavior from PyTorch

As suggested in the [docstring of the IterableDataset class](https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset), we could pass a `worker_init_fn` to the DataLoader to fix this. It could be called `streaming_worker_init_fn` for example.

However, while this solution works, I'm worried that many users simply don't know about this parameter and just start their training with duplicate data without knowing it. That's why I'm more in favor of integrating the check on the worker id directly in `datasets` in our implementation of `IterableDataset.__iter__`.
2021-12-14T10:52:17Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-993417640
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps47Nlmo
https://api.github.com/repos/huggingface/datasets/issues/comments/993417640/reactions
2021-12-14T10:53:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/993417640
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Fixed by https://github.com/huggingface/datasets/pull/4375
2022-07-06T14:15:43Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1176277535
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5GHJIf
https://api.github.com/repos/huggingface/datasets/issues/comments/1176277535/reactions
2022-07-06T14:15:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1176277535
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Fixed by #4375

Thanks!
2022-08-08T09:22:28Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1207878759
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5H_sRn
https://api.github.com/repos/huggingface/datasets/issues/comments/1207878759/reactions
2022-08-08T09:22:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1207878759
https://avatars.githubusercontent.com/u/16486492?v=4
https://api.github.com/users/cloudyuyuyu/events{/privacy}
https://api.github.com/users/cloudyuyuyu/followers
https://api.github.com/users/cloudyuyuyu/following{/other_user}
https://api.github.com/users/cloudyuyuyu/gists{/gist_id}
https://github.com/cloudyuyuyu
cloudyuyuyu
MDQ6VXNlcjE2NDg2NDky
https://api.github.com/users/cloudyuyuyu/orgs
https://api.github.com/users/cloudyuyuyu/received_events
https://api.github.com/users/cloudyuyuyu/repos
https://api.github.com/users/cloudyuyuyu/starred{/owner}{/repo}
https://api.github.com/users/cloudyuyuyu/subscriptions
User
https://api.github.com/users/cloudyuyuyu
public
NONE
Hi there @lhoestq @cloudyuyuyu 
I met that problem recently, and #4375 is really useful because I finally found out I am training with duplicate data.
However, in multi-GPU training, I'm using DDP mode and IterableDataset, which still yields duplicate data for each progress. And this is dangerous because users maybe not realize this behavior.
2022-10-18T07:56:56Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1281965683
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5MaT5z
https://api.github.com/repos/huggingface/datasets/issues/comments/1281965683/reactions
2022-10-18T07:56:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1281965683
https://avatars.githubusercontent.com/u/9592150?v=4
https://api.github.com/users/Ethan-yt/events{/privacy}
https://api.github.com/users/Ethan-yt/followers
https://api.github.com/users/Ethan-yt/following{/other_user}
https://api.github.com/users/Ethan-yt/gists{/gist_id}
https://github.com/Ethan-yt
Ethan-yt
MDQ6VXNlcjk1OTIxNTA=
https://api.github.com/users/Ethan-yt/orgs
https://api.github.com/users/Ethan-yt/received_events
https://api.github.com/users/Ethan-yt/repos
https://api.github.com/users/Ethan-yt/starred{/owner}{/repo}
https://api.github.com/users/Ethan-yt/subscriptions
User
https://api.github.com/users/Ethan-yt
public
MEMBER
If the worker_info.id is unique per process it should work fine, could you check that they're unique ?

The code to get the worker_info in each worker is `torch.utils.data.get_worker_info()`
2022-10-18T17:27:10Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1282758859
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5MdVjL
https://api.github.com/repos/huggingface/datasets/issues/comments/1282758859/reactions
2022-10-18T17:28:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1282758859
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
test.py
```python
import json
import os

import torch
from torch.utils.data import IterableDataset, DataLoader
from transformers import PreTrainedTokenizer, TrainingArguments

from common.arguments import DataTrainingArguments, ModelArguments


class MyIterableDataset(IterableDataset):
    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        print(worker_info)
        return iter(range(3))


if __name__ == '__main__':
    dataset = MyIterableDataset()
    dataloader = DataLoader(dataset, num_workers=1)
    for i in dataloader:
        print(i)

```


```sh
$ python3 -m torch.distributed.launch \
  --nproc_per_node=2 test.py
WorkerInfo(id=0, num_workers=1, seed=5545685212307804959, dataset=<__main__.MyIterableDataset object at 0x7f92648cf6a0>)
WorkerInfo(id=0, num_workers=1, seed=3174108029709729025, dataset=<__main__.MyIterableDataset object at 0x7f19ab961670>)
tensor([0])
tensor([1])
tensor([2])
tensor([0])
tensor([1])
tensor([2])
```

@lhoestq they are not unique
2022-10-28T11:15:51Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1294870703
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5NLiiv
https://api.github.com/repos/huggingface/datasets/issues/comments/1294870703/reactions
2022-10-28T11:16:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1294870703
https://avatars.githubusercontent.com/u/9592150?v=4
https://api.github.com/users/Ethan-yt/events{/privacy}
https://api.github.com/users/Ethan-yt/followers
https://api.github.com/users/Ethan-yt/following{/other_user}
https://api.github.com/users/Ethan-yt/gists{/gist_id}
https://github.com/Ethan-yt
Ethan-yt
MDQ6VXNlcjk1OTIxNTA=
https://api.github.com/users/Ethan-yt/orgs
https://api.github.com/users/Ethan-yt/received_events
https://api.github.com/users/Ethan-yt/repos
https://api.github.com/users/Ethan-yt/starred{/owner}{/repo}
https://api.github.com/users/Ethan-yt/subscriptions
User
https://api.github.com/users/Ethan-yt
public
MEMBER
It looks like a bug from pytorch no ? How can we know which data should go in which process when using DDP ?

I guess we need to check `torch.distributed.get_world_size()` and `torch.distributed.get_rank()` as well. Not fan of the design here tbh, but that's how it is
2022-10-28T13:13:03Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1294985029
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5NL-dF
https://api.github.com/repos/huggingface/datasets/issues/comments/1294985029/reactions
2022-10-28T13:20:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1294985029
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> It looks like a bug from pytorch no ? How can we know which data should go in which process when using DDP ?
> 
> I guess we need to check `torch.distributed.get_world_size()` and `torch.distributed.get_rank()` as well. Not fan of the design here tbh, but that's how it is

Maybe we should document it?
2022-10-31T02:46:03Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1296467468
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5NRoYM
https://api.github.com/repos/huggingface/datasets/issues/comments/1296467468/reactions
2022-11-18T03:42:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1296467468
https://avatars.githubusercontent.com/u/9592150?v=4
https://api.github.com/users/Ethan-yt/events{/privacy}
https://api.github.com/users/Ethan-yt/followers
https://api.github.com/users/Ethan-yt/following{/other_user}
https://api.github.com/users/Ethan-yt/gists{/gist_id}
https://github.com/Ethan-yt
Ethan-yt
MDQ6VXNlcjk1OTIxNTA=
https://api.github.com/users/Ethan-yt/orgs
https://api.github.com/users/Ethan-yt/received_events
https://api.github.com/users/Ethan-yt/repos
https://api.github.com/users/Ethan-yt/starred{/owner}{/repo}
https://api.github.com/users/Ethan-yt/subscriptions
User
https://api.github.com/users/Ethan-yt
public
NONE
Never mind. After reading the code, `IterableDatasetShard` has solved this problem.
2022-11-18T03:43:45Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1319504654
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5OpgsO
https://api.github.com/repos/huggingface/datasets/issues/comments/1319504654/reactions
2022-11-18T03:43:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1319504654
https://avatars.githubusercontent.com/u/9592150?v=4
https://api.github.com/users/Ethan-yt/events{/privacy}
https://api.github.com/users/Ethan-yt/followers
https://api.github.com/users/Ethan-yt/following{/other_user}
https://api.github.com/users/Ethan-yt/gists{/gist_id}
https://github.com/Ethan-yt
Ethan-yt
MDQ6VXNlcjk1OTIxNTA=
https://api.github.com/users/Ethan-yt/orgs
https://api.github.com/users/Ethan-yt/received_events
https://api.github.com/users/Ethan-yt/repos
https://api.github.com/users/Ethan-yt/starred{/owner}{/repo}
https://api.github.com/users/Ethan-yt/subscriptions
User
https://api.github.com/users/Ethan-yt
public
MEMBER
I'm re-opening this one since I think it should be supported by `datasets` natively
2022-12-14T16:03:34Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1351692730
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5QkTG6
https://api.github.com/repos/huggingface/datasets/issues/comments/1351692730/reactions
2022-12-14T16:03:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1351692730
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
hmm actually let me open a new issue on DDP - original post was for single node
2022-12-14T16:04:22Z
https://github.com/huggingface/datasets/issues/3423#issuecomment-1351693908
https://api.github.com/repos/huggingface/datasets/issues/3423
IC_kwDODunzps5QkTZU
https://api.github.com/repos/huggingface/datasets/issues/comments/1351693908/reactions
2022-12-14T16:04:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1351693908
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug
The data is repeated num_works times when we load_dataset with streaming and set num_works > 1 when construct dataloader

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import pandas as pd
import numpy as np
import os

from datasets import load_dataset
from torch.utils.data import DataLoader
from tqdm import tqdm
import shutil

NUM_OF_USER = 1000000
NUM_OF_ACTION = 50000
NUM_OF_SEQUENCE = 10000
NUM_OF_FILES = 32
NUM_OF_WORKERS = 16

if __name__ == "__main__":
    shutil.rmtree("./dataset")
    for i in range(NUM_OF_FILES):
        sequence_data = pd.DataFrame(
            {
                "imei": np.random.randint(1, NUM_OF_USER, size=NUM_OF_SEQUENCE),
                "sequence": np.random.randint(1, NUM_OF_ACTION, size=NUM_OF_SEQUENCE)
            }
        )

        if not os.path.exists("./dataset"):
            os.makedirs("./dataset")

        sequence_data.to_csv(f"./dataset/sequence_data_{i}.csv",
                            
 index=False)

    dataset = load_dataset("csv",
                           data_files=[os.path.join("./dataset",file) for file in os.listdir("./dataset") if file.endswith(".csv")],
                           split="train",
                           streaming=True).with_format("torch")
    data_loader = DataLoader(dataset,
                             batch_size=1024,
                             num_workers=NUM_OF_WORKERS)
    
    result = pd.DataFrame()
    for i, batch in tqdm(enumerate(data_loader)):
        result = pd.concat([result, 
                           pd.DataFrame(batch)],
                           axis=0)
    result.to_csv(f"num_work_{NUM_OF_WORKERS}.csv", index=False)

```

## Expected results
data do not duplicate 

## Actual results
data duplicate NUM_OF_WORKERS = 16 
![image](https://user-images.githubusercontent.com/16486492/145748707-9d2df25b-2f4f-4d7b-a83e-242be4fc8934.png)


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:datasets==1.14.0
- Platform:transformers==4.11.3
- Python version:3.8
- PyArrow version:
