DataLoader(datasets) become more and more slowly within iterations
MEMBER
Hi ! Thanks for reporting.
Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
It would be nice to know whether it comes from the dataloader or not
2020-11-24T09:31:08Z
https://github.com/huggingface/datasets/issues/877#issuecomment-732772900
https://api.github.com/repos/huggingface/datasets/issues/877
MDEyOklzc3VlQ29tbWVudDczMjc3MjkwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/732772900/reactions
2020-11-24T09:31:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732772900
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Hi ! Thanks for reporting.
> Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
> It would be nice to know whether it comes from the dataloader or not

I did not iter data from raw dataset, maybe I will test later. Now I iter all files directly from `open(file)`,  around 20000it/s.
2020-11-26T02:05:29Z
https://github.com/huggingface/datasets/issues/877#issuecomment-734030603
https://api.github.com/repos/huggingface/datasets/issues/877
MDEyOklzc3VlQ29tbWVudDczNDAzMDYwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/734030603/reactions
2020-11-26T02:05:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734030603
https://avatars.githubusercontent.com/u/25664170?v=4
https://api.github.com/users/shexuan/events{/privacy}
https://api.github.com/users/shexuan/followers
https://api.github.com/users/shexuan/following{/other_user}
https://api.github.com/users/shexuan/gists{/gist_id}
https://github.com/shexuan
shexuan
MDQ6VXNlcjI1NjY0MTcw
https://api.github.com/users/shexuan/orgs
https://api.github.com/users/shexuan/received_events
https://api.github.com/users/shexuan/repos
https://api.github.com/users/shexuan/starred{/owner}{/repo}
https://api.github.com/users/shexuan/subscriptions
User
https://api.github.com/users/shexuan
public
NONE
> > Hi ! Thanks for reporting.
> > Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
> > It would be nice to know whether it comes from the dataloader or not
> 
> I did not iter data from raw dataset, maybe I will test later. Now I iter all files directly from `open(file)`, around 20000it/s.

Hi ! I meet the same problem, how did you solve it?
2024-11-22T03:02:52Z
https://github.com/huggingface/datasets/issues/877#issuecomment-2492784804
https://api.github.com/repos/huggingface/datasets/issues/877
IC_kwDODunzps6UlOCk
https://api.github.com/repos/huggingface/datasets/issues/comments/2492784804/reactions
2024-11-22T03:02:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2492784804
https://avatars.githubusercontent.com/u/30164440?v=4
https://api.github.com/users/NOTGOOOOD/events{/privacy}
https://api.github.com/users/NOTGOOOOD/followers
https://api.github.com/users/NOTGOOOOD/following{/other_user}
https://api.github.com/users/NOTGOOOOD/gists{/gist_id}
https://github.com/NOTGOOOOD
NOTGOOOOD
MDQ6VXNlcjMwMTY0NDQw
https://api.github.com/users/NOTGOOOOD/orgs
https://api.github.com/users/NOTGOOOOD/received_events
https://api.github.com/users/NOTGOOOOD/repos
https://api.github.com/users/NOTGOOOOD/starred{/owner}{/repo}
https://api.github.com/users/NOTGOOOOD/subscriptions
User
https://api.github.com/users/NOTGOOOOD
public
Hello, when I for loop my dataloader, the loading speed is becoming more and more slowly!
```
dataset = load_from_disk(dataset_path)  # around 21,000,000 lines

lineloader = tqdm(DataLoader(dataset, batch_size=1))
for idx, line in enumerate(lineloader):
     # do some thing for each line
```
In the begining, the loading speed is around 2000it/s, but after 1 minutes later, the speed is much slower, just around 800it/s.

And when I set `num_workers=4` in DataLoader, the loading speed is much lower, just 130it/s.

Could you please help me with this problem?
Thanks a lot!