Make `interleave_datasets` more robust
NONE
Hi @lhoestq  Any response on this issue?
2022-01-25T17:46:35Z
https://github.com/huggingface/datasets/issues/3064#issuecomment-1021449796
https://api.github.com/repos/huggingface/datasets/issues/3064
IC_kwDODunzps484hZE
https://api.github.com/repos/huggingface/datasets/issues/comments/1021449796/reactions
2022-01-25T17:46:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1021449796
https://avatars.githubusercontent.com/u/32699797?v=4
https://api.github.com/users/sbmaruf/events{/privacy}
https://api.github.com/users/sbmaruf/followers
https://api.github.com/users/sbmaruf/following{/other_user}
https://api.github.com/users/sbmaruf/gists{/gist_id}
https://github.com/sbmaruf
sbmaruf
MDQ6VXNlcjMyNjk5Nzk3
https://api.github.com/users/sbmaruf/orgs
https://api.github.com/users/sbmaruf/received_events
https://api.github.com/users/sbmaruf/repos
https://api.github.com/users/sbmaruf/starred{/owner}{/repo}
https://api.github.com/users/sbmaruf/subscriptions
User
https://api.github.com/users/sbmaruf
public
MEMBER
Hi ! Sorry for the late response

I agree `interleave_datasets` would benefit a lot from having more flexibility. If I understand correctly it would be nice to be able to define stopping strategies like `stop="first_exhausted"` (default) or `stop="all_exhausted"`. If you'd like to contribute this feature I'd be happy to give you some pointers :)

Also one can already set the max number of iterations per dataset by doing `dataset.take(n)` on the dataset that should only have `n` samples.

Regarding the `iter_cnt` counter, I think this requires a bit more thoughts, since we might have to be able to backpropagate the the counter if `map` or other transforms have been applied after `interleave_datasets`. 
2022-01-26T15:51:47Z
https://github.com/huggingface/datasets/issues/3064#issuecomment-1022333385
https://api.github.com/repos/huggingface/datasets/issues/3064
IC_kwDODunzps4875HJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1022333385/reactions
2022-01-26T15:51:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1022333385
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@sbmaruf I just notice that (1)`interleave_datasets` only samples indices once and reuse for all epochs, and (2) it's limited by the smallest dataset. Do you figure out an alternative way to achieve the same purpose?
2022-07-30T08:47:26Z
https://github.com/huggingface/datasets/issues/3064#issuecomment-1200118801
https://api.github.com/repos/huggingface/datasets/issues/3064
IC_kwDODunzps5HiFwR
https://api.github.com/repos/huggingface/datasets/issues/comments/1200118801/reactions
2022-07-30T08:47:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1200118801
https://avatars.githubusercontent.com/u/4197249?v=4
https://api.github.com/users/memray/events{/privacy}
https://api.github.com/users/memray/followers
https://api.github.com/users/memray/following{/other_user}
https://api.github.com/users/memray/gists{/gist_id}
https://github.com/memray
memray
MDQ6VXNlcjQxOTcyNDk=
https://api.github.com/users/memray/orgs
https://api.github.com/users/memray/received_events
https://api.github.com/users/memray/repos
https://api.github.com/users/memray/starred{/owner}{/repo}
https://api.github.com/users/memray/subscriptions
User
https://api.github.com/users/memray
public
**Is your feature request related to a problem? Please describe.**
Right now there are few hiccups using `interleave_datasets`. Interleaved dataset iterates until the smallest dataset completes it's iterator. In this way larger datasets may not complete full epoch of iteration. 
It creates new problems in calculation of epoch since there are no way to track which dataset from `interleave_datasets` completes how many epoch.

**Describe the solution you'd like**
For `interleave_datasets` module, 
- [ ] Add a boolean argument `--stop-iter` in `interleave_datasets` that enables dataset to either iterate infinite amount of time or not. That means it should not return `StopIterator` exception in case `--stop-iter=False`.
- [ ] Internal list variable `iter_cnt` that explains how many times (in steps/epochs) each dataset iterates at a given point.
- [ ] Add an argument `--max-iter` (list type) that explain maximum amount of time each of the dataset can iterate. After complete  `--max-iter` of one dataset, other dataset should continue sampling and when all the dataset finish their respective `--max-iter`, only then return `StopIterator`

Note: I'm new to `datasets` api. May be these features are already there in the datasets. 

Since multitask training is the latest trends, I believe this feature would make the `datasets` api more popular.

@lhoestq 