map(...batch_size=None) for IterableDataset
COLLABORATOR
Hi! I agree it makes sense for `IterableDataset.map` to support the `batch_size=None` case. This should be super easy to fix.
2022-12-01T17:36:49Z
https://github.com/huggingface/datasets/issues/5325#issuecomment-1334119953
https://api.github.com/repos/huggingface/datasets/issues/5325
IC_kwDODunzps5PhQ4R
https://api.github.com/repos/huggingface/datasets/issues/comments/1334119953/reactions
2022-12-01T17:36:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1334119953
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
@mariosasko as this is something simple maybe I can include it as part of https://github.com/huggingface/datasets/pull/5311? Let me know :+1:
2022-12-05T08:55:40Z
https://github.com/huggingface/datasets/issues/5325#issuecomment-1336977215
https://api.github.com/repos/huggingface/datasets/issues/5325
IC_kwDODunzps5PsKc_
https://api.github.com/repos/huggingface/datasets/issues/comments/1336977215/reactions
2022-12-05T08:55:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1336977215
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
#self-assign
2022-12-06T15:52:15Z
https://github.com/huggingface/datasets/issues/5325#issuecomment-1339586619
https://api.github.com/repos/huggingface/datasets/issues/5325
IC_kwDODunzps5P2Hg7
https://api.github.com/repos/huggingface/datasets/issues/comments/1339586619/reactions
2022-12-06T15:52:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1339586619
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
Feel free to close this @lhoestq as part of https://github.com/huggingface/datasets/pull/5336 :hugs:
2022-12-07T15:28:11Z
https://github.com/huggingface/datasets/issues/5325#issuecomment-1341134391
https://api.github.com/repos/huggingface/datasets/issues/5325
IC_kwDODunzps5P8BY3
https://api.github.com/repos/huggingface/datasets/issues/comments/1341134391/reactions
2022-12-07T15:28:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1341134391
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
Thanks again :)

> For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

This is interesting as well, if anyone wants to explore
2022-12-07T15:54:42Z
https://github.com/huggingface/datasets/issues/5325#issuecomment-1341172528
https://api.github.com/repos/huggingface/datasets/issues/5325
IC_kwDODunzps5P8Ksw
https://api.github.com/repos/huggingface/datasets/issues/comments/1341172528/reactions
2022-12-07T15:54:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1341172528
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Feature request

Dataset.map(...) allows batch_size to be None. It would be nice if IterableDataset did too.

### Motivation

Although it may seem a bit of a spurious request given that `IterableDataset` is meant for larger than memory datasets, but there are a couple of reasons why this might be nice.

One is that load_dataset(...) can return either IterableDataset or Dataset. mypy will then complain if batch_size=None even if we know it is Dataset. Of course we can do:

    assert isinstance(d, datasets.DatasetDict)

But it is a mild inconvenience. What's more annoying is that whenever we use something like e.g. `combine_datasets(...)`, we end up with the union again, and so have to do the assert again.

Another is that we could actually end up with an IterableDataset small enough for memory in normal/correct usage, e.g. by filtering a massive IterableDataset.

For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

### Your contribution

Not this time.