Multiple dataloader memory error
COLLABORATOR
Hi!

200+ data loaders is a lot. Have you tried to reduce the number of datasets by concatenating/interleaving the ones with the same structure/task (the API is `{concatenate_datasets/interleave_datasets}([dset1, ..., dset_N])`)?
2022-09-08T16:36:58Z
https://github.com/huggingface/datasets/issues/4873#issuecomment-1240955745
https://api.github.com/repos/huggingface/datasets/issues/4873
IC_kwDODunzps5J93th
https://api.github.com/repos/huggingface/datasets/issues/comments/1240955745/reactions
2022-09-08T16:37:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1240955745
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hi @mariosasko, thank you for your reply. I tried pre-concatenating different datasets into one, but one key need is to keep each batch the same data type. Considering that the concatenate-then-segment operation for prefetched samples may span across different data types after concatenating/interleaving (cuz different data sources are mixed), any solution to remain the same data source for each batch?
2022-09-09T03:01:28Z
https://github.com/huggingface/datasets/issues/4873#issuecomment-1241443032
https://api.github.com/repos/huggingface/datasets/issues/4873
IC_kwDODunzps5J_urY
https://api.github.com/repos/huggingface/datasets/issues/comments/1241443032/reactions
2022-09-09T03:02:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1241443032
https://avatars.githubusercontent.com/u/13767887?v=4
https://api.github.com/users/cyk1337/events{/privacy}
https://api.github.com/users/cyk1337/followers
https://api.github.com/users/cyk1337/following{/other_user}
https://api.github.com/users/cyk1337/gists{/gist_id}
https://github.com/cyk1337
cyk1337
MDQ6VXNlcjEzNzY3ODg3
https://api.github.com/users/cyk1337/orgs
https://api.github.com/users/cyk1337/received_events
https://api.github.com/users/cyk1337/repos
https://api.github.com/users/cyk1337/starred{/owner}{/repo}
https://api.github.com/users/cyk1337/subscriptions
User
https://api.github.com/users/cyk1337
public
NONE
@cyk1337 have you found any solutions to it?
@mariosasko I tried with interleave_datasets to sample batches from two large datasets (wikipedia alike) and it results in out-of-memory error during data loading (16gpus, >1TB physical memory). Do you have any idea about it?
2023-01-26T02:01:11Z
https://github.com/huggingface/datasets/issues/4873#issuecomment-1404473793
https://api.github.com/repos/huggingface/datasets/issues/4873
IC_kwDODunzps5TtpHB
https://api.github.com/repos/huggingface/datasets/issues/comments/1404473793/reactions
2023-01-26T02:01:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1404473793
https://avatars.githubusercontent.com/u/4197249?v=4
https://api.github.com/users/memray/events{/privacy}
https://api.github.com/users/memray/followers
https://api.github.com/users/memray/following{/other_user}
https://api.github.com/users/memray/gists{/gist_id}
https://github.com/memray
memray
MDQ6VXNlcjQxOTcyNDk=
https://api.github.com/users/memray/orgs
https://api.github.com/users/memray/received_events
https://api.github.com/users/memray/repos
https://api.github.com/users/memray/starred{/owner}{/repo}
https://api.github.com/users/memray/subscriptions
User
https://api.github.com/users/memray
public
For the use of multiple datasets and tasks, we use around more than 200+ dataloaders, then pass it into `dataloader1, dataloader2, ..., dataloader200=accelerate.prepare(dataloader1, dataloader2, ..., dataloader200)`
It causes the memory error when generating batches. Any solutions to it?

```bash
  File "/home/xxx/my_code/src/utils/data_utils.py", line 54, in generate_batch
    x = next(iterator)
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/accelerate/data_loader.py", line 301, in __iter__
    for batch in super().__iter__():
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 28, in fetch
    data.append(next(self.dataset_iter))
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/accelerate/data_loader.py", line 249, in __iter__
    for element in self.dataset:
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 503, in __iter__
    for key, example in self._iter():
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 500, in _iter
    yield from ex_iterable
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 231, in __iter__
    new_key = "_".join(str(key) for key in keys)
MemoryError
```