Calling shuffle on IterableDataset will disable batching in case any functions were mapped
MEMBER
Hi :) Good catch ! Feel free to open a PR if you want to contribute, this would be very welcome ;)
2021-07-26T14:08:11Z
https://github.com/huggingface/datasets/issues/2716#issuecomment-886735769
https://api.github.com/repos/huggingface/datasets/issues/2716
IC_kwDODunzps402oOZ
https://api.github.com/repos/huggingface/datasets/issues/comments/886735769/reactions
2021-07-26T14:08:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/886735769
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Have raised the PR [here](https://github.com/huggingface/datasets/pull/2717)
2021-07-26T14:46:46Z
https://github.com/huggingface/datasets/issues/2716#issuecomment-886767889
https://api.github.com/repos/huggingface/datasets/issues/2716
IC_kwDODunzps402wER
https://api.github.com/repos/huggingface/datasets/issues/comments/886767889/reactions
2021-07-26T14:46:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/886767889
https://avatars.githubusercontent.com/u/7098967?v=4
https://api.github.com/users/amankhandelia/events{/privacy}
https://api.github.com/users/amankhandelia/followers
https://api.github.com/users/amankhandelia/following{/other_user}
https://api.github.com/users/amankhandelia/gists{/gist_id}
https://github.com/amankhandelia
amankhandelia
MDQ6VXNlcjcwOTg5Njc=
https://api.github.com/users/amankhandelia/orgs
https://api.github.com/users/amankhandelia/received_events
https://api.github.com/users/amankhandelia/repos
https://api.github.com/users/amankhandelia/starred{/owner}{/repo}
https://api.github.com/users/amankhandelia/subscriptions
User
https://api.github.com/users/amankhandelia
public
MEMBER
Fixed by #2717.
2021-07-26T18:04:43Z
https://github.com/huggingface/datasets/issues/2716#issuecomment-886913373
https://api.github.com/repos/huggingface/datasets/issues/2716
IC_kwDODunzps403Tld
https://api.github.com/repos/huggingface/datasets/issues/comments/886913373/reactions
2021-07-26T18:04:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/886913373
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
When using dataset in streaming mode, if one applies `shuffle` method on the dataset and `map` method for which `batched=True` than the batching operation will not happen, instead `batched` will be set to `False`

I did RCA on the dataset codebase, the problem is emerging from [this line of code](https://github.com/huggingface/datasets/blob/d25a0bf94d9f9a9aa6cabdf5b450b9c327d19729/src/datasets/iterable_dataset.py#L197) here as it is
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batch_size=self.batch_size`, as one can see it is missing batched argument, which means that the iterator fallsback to default constructor value, which in this case is `False`.
To remedy the problem we can change this line to
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batched=self.batched, batch_size=self.batch_size`
