sample multiple datasets 
NONE
here I share my dataloader currently for multiple tasks: https://gist.github.com/rabeehkarimimahabadi/39f9444a4fb6f53dcc4fca5d73bf8195 

I need to train my model distributedly with this dataloader, "MultiTasksataloader", currently this does not work in distributed fasion,
to save on memory I tried to use iterative datasets, could you have a look in this dataloader and tell me if this is indeed the case? not sure how to make datasets being iterative to not load them in memory, then I remove the sampler for dataloader, and shard the data per core, could you tell me please how I should implement this case in datasets library? and how do you find my implementation in terms of correctness? thanks 

2020-12-02T01:32:44Z
https://github.com/huggingface/datasets/issues/961#issuecomment-736928623
https://api.github.com/repos/huggingface/datasets/issues/961
MDEyOklzc3VlQ29tbWVudDczNjkyODYyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/736928623/reactions
2020-12-02T01:32:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/736928623
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
NONE
Hi @rabeehkarimimahabadi any luck with updating the multi-task data loader to work with distributed training?
2023-05-01T08:36:22Z
https://github.com/huggingface/datasets/issues/961#issuecomment-1529485245
https://api.github.com/repos/huggingface/datasets/issues/961
IC_kwDODunzps5bKhe9
https://api.github.com/repos/huggingface/datasets/issues/comments/1529485245/reactions
2023-05-01T08:36:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1529485245
https://avatars.githubusercontent.com/u/21266230?v=4
https://api.github.com/users/pushkalkatara/events{/privacy}
https://api.github.com/users/pushkalkatara/followers
https://api.github.com/users/pushkalkatara/following{/other_user}
https://api.github.com/users/pushkalkatara/gists{/gist_id}
https://github.com/pushkalkatara
pushkalkatara
MDQ6VXNlcjIxMjY2MjMw
https://api.github.com/users/pushkalkatara/orgs
https://api.github.com/users/pushkalkatara/received_events
https://api.github.com/users/pushkalkatara/repos
https://api.github.com/users/pushkalkatara/starred{/owner}{/repo}
https://api.github.com/users/pushkalkatara/subscriptions
User
https://api.github.com/users/pushkalkatara
public
CONTRIBUTOR
Hi @pushkalkatara yes I solved it back then, here please find my implementation https://github.com/rabeehk/hyperformer/blob/main/hyperformer/data/multitask_sampler.py 
2023-05-01T11:59:18Z
https://github.com/huggingface/datasets/issues/961#issuecomment-1529619797
https://api.github.com/repos/huggingface/datasets/issues/961
IC_kwDODunzps5bLCVV
https://api.github.com/repos/huggingface/datasets/issues/comments/1529619797/reactions
2023-05-01T11:59:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1529619797
https://avatars.githubusercontent.com/u/6278280?v=4
https://api.github.com/users/rabeehk/events{/privacy}
https://api.github.com/users/rabeehk/followers
https://api.github.com/users/rabeehk/following{/other_user}
https://api.github.com/users/rabeehk/gists{/gist_id}
https://github.com/rabeehk
rabeehk
MDQ6VXNlcjYyNzgyODA=
https://api.github.com/users/rabeehk/orgs
https://api.github.com/users/rabeehk/received_events
https://api.github.com/users/rabeehk/repos
https://api.github.com/users/rabeehk/starred{/owner}{/repo}
https://api.github.com/users/rabeehk/subscriptions
User
https://api.github.com/users/rabeehk
public
NONE
Thanks @rabeehk for sharing. 

The sampler basically returns a list of integers to sample from each task's dataset. I was wondering how to use it with two `torch.Dataset` of different tasks. Also, do I need to shard across processes while creating an Iterable Dataset?

2023-05-08T17:05:39Z
https://github.com/huggingface/datasets/issues/961#issuecomment-1538735021
https://api.github.com/repos/huggingface/datasets/issues/961
IC_kwDODunzps5btzut
https://api.github.com/repos/huggingface/datasets/issues/comments/1538735021/reactions
2023-05-08T17:05:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1538735021
https://avatars.githubusercontent.com/u/21266230?v=4
https://api.github.com/users/pushkalkatara/events{/privacy}
https://api.github.com/users/pushkalkatara/followers
https://api.github.com/users/pushkalkatara/following{/other_user}
https://api.github.com/users/pushkalkatara/gists{/gist_id}
https://github.com/pushkalkatara
pushkalkatara
MDQ6VXNlcjIxMjY2MjMw
https://api.github.com/users/pushkalkatara/orgs
https://api.github.com/users/pushkalkatara/received_events
https://api.github.com/users/pushkalkatara/repos
https://api.github.com/users/pushkalkatara/starred{/owner}{/repo}
https://api.github.com/users/pushkalkatara/subscriptions
User
https://api.github.com/users/pushkalkatara
public
COLLABORATOR
We now have `interleave_datasets` in the API that allows you to cycle/sample with probabilities (with various stopping strategies) through a list of datasets. However, more specific behavior should be implemented manually.
2023-07-20T14:08:57Z
https://github.com/huggingface/datasets/issues/961#issuecomment-1643996875
https://api.github.com/repos/huggingface/datasets/issues/961
IC_kwDODunzps5h_WbL
https://api.github.com/repos/huggingface/datasets/issues/comments/1643996875/reactions
2023-07-20T14:08:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1643996875
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hi @mariosasko, @pushkalkatara 

I have multi dataset for ASR task. I have multilingual such as: English, China, Japan, Vietnamese. Each dataset is loaded as dataset and they are imbalance dataset. Now I training, I want to equal sampler, it means each batch loader in training has equal number sampler in each language.

For example: Batch_size = 16:
English sample:4
China sample:4
Japan sample:4
Vietnamese sample:4 

How can I do it? I saw that  interleave_datasets only splits data with probability an concatenates all of them after that, it not affect in training data loader. Forgiving me if I wrong.

Thank you for your help.
2024-06-14T08:20:31Z
https://github.com/huggingface/datasets/issues/961#issuecomment-2167508579
https://api.github.com/repos/huggingface/datasets/issues/961
IC_kwDODunzps6BMY5j
https://api.github.com/repos/huggingface/datasets/issues/comments/2167508579/reactions
2024-06-17T08:23:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2167508579
https://avatars.githubusercontent.com/u/42369268?v=4
https://api.github.com/users/phamkhactu/events{/privacy}
https://api.github.com/users/phamkhactu/followers
https://api.github.com/users/phamkhactu/following{/other_user}
https://api.github.com/users/phamkhactu/gists{/gist_id}
https://github.com/phamkhactu
phamkhactu
MDQ6VXNlcjQyMzY5MjY4
https://api.github.com/users/phamkhactu/orgs
https://api.github.com/users/phamkhactu/received_events
https://api.github.com/users/phamkhactu/repos
https://api.github.com/users/phamkhactu/starred{/owner}{/repo}
https://api.github.com/users/phamkhactu/subscriptions
User
https://api.github.com/users/phamkhactu
public
Hi
I am dealing with multiple datasets, I need to have a dataloader over them with a condition that in each batch data samples are coming from one of the datasets. My main question is: 
-  I need to have a way to sample the datasets first with some weights, lets say 2x dataset1 1x dataset2, could you point me how I can do it

sub-questions:
- I want to concat sampled datasets and define one dataloader on it, then I need a way to make sure batches come from 1 dataset in each iteration, could you assist me how I can do?
- I use iterative-type of datasets, but I need a method of shuffling still since it brings accuracy performance issues if not doing it, thanks for the help. 