Fix iter_batches
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-10-14T12:14:03Z
https://github.com/huggingface/datasets/pull/5115#issuecomment-1278925635
https://api.github.com/repos/huggingface/datasets/issues/5115
IC_kwDODunzps5MOttD
https://api.github.com/repos/huggingface/datasets/issues/comments/1278925635/reactions
2022-10-14T15:02:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278925635
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
I also ran the code in https://github.com/huggingface/datasets/issues/5111 and it works fine now :)
2022-10-14T13:39:46Z
https://github.com/huggingface/datasets/pull/5115#issuecomment-1279024645
https://api.github.com/repos/huggingface/datasets/issues/5115
IC_kwDODunzps5MPF4F
https://api.github.com/repos/huggingface/datasets/issues/comments/1279024645/reactions
2022-10-14T13:39:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1279024645
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
This is ready for review :)
2022-10-14T14:04:47Z
https://github.com/huggingface/datasets/pull/5115#issuecomment-1279055329
https://api.github.com/repos/huggingface/datasets/issues/5115
IC_kwDODunzps5MPNXh
https://api.github.com/repos/huggingface/datasets/issues/comments/1279055329/reactions
2022-10-14T14:04:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1279055329
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
The `pa.Table.to_reader()` method available in `pyarrow>=8.0.0` may return chunks of size < `max_chunksize`, therefore `iter_batches` can return batches smaller than the `batch_size` specified by the user

Therefore batched `map` couldn't always use batches of the right size, e.g. this fails because it runs only on one batch of one element:
```python

from datasets import Dataset, concatenate_datasets

ds = concatenate_datasets([Dataset.from_dict({"a": [i]}) for i in range(10)])

ds2 = ds.map(lambda _: {}, batched=True)
assert list(ds2) == list(ds)
```

This was introduced in https://github.com/huggingface/datasets/pull/5030

Close https://github.com/huggingface/datasets/issues/5111

This will require a patch release along with https://github.com/huggingface/datasets/pull/5113

TODO:
- [x] fix tests
- [x] add more tests