when using to_tf_dataset with shuffle is true, not all completed batches are made
MEMBER
Maybe @Rocketknight1 can help here
2022-03-08T00:11:41Z
https://github.com/huggingface/datasets/issues/3831#issuecomment-1061270871
https://api.github.com/repos/huggingface/datasets/issues/3831
IC_kwDODunzps4_QbVX
https://api.github.com/repos/huggingface/datasets/issues/comments/1061270871/reactions
2022-03-08T00:11:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1061270871
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Hi @greenned, this is expected behaviour for `to_tf_dataset`. By default, we drop the smaller 'remainder' batch during training (i.e. when `shuffle=True`). If you really want to keep that batch, you can set `drop_remainder=False` when calling `to_tf_dataset()`.
2022-03-08T12:18:52Z
https://github.com/huggingface/datasets/issues/3831#issuecomment-1061718719
https://api.github.com/repos/huggingface/datasets/issues/3831
IC_kwDODunzps4_SIq_
https://api.github.com/repos/huggingface/datasets/issues/comments/1061718719/reactions
2022-03-08T12:18:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1061718719
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
NONE
@Rocketknight1 Oh, thank you. I didn't get **drop_remainder** Have a nice day!
2022-03-08T12:52:10Z
https://github.com/huggingface/datasets/issues/3831#issuecomment-1061745122
https://api.github.com/repos/huggingface/datasets/issues/3831
IC_kwDODunzps4_SPHi
https://api.github.com/repos/huggingface/datasets/issues/comments/1061745122/reactions
2022-03-08T12:52:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1061745122
https://avatars.githubusercontent.com/u/56626346?v=4
https://api.github.com/users/bv-sm-yoon/events{/privacy}
https://api.github.com/users/bv-sm-yoon/followers
https://api.github.com/users/bv-sm-yoon/following{/other_user}
https://api.github.com/users/bv-sm-yoon/gists{/gist_id}
https://github.com/bv-sm-yoon
bv-sm-yoon
MDQ6VXNlcjU2NjI2MzQ2
https://api.github.com/users/bv-sm-yoon/orgs
https://api.github.com/users/bv-sm-yoon/received_events
https://api.github.com/users/bv-sm-yoon/repos
https://api.github.com/users/bv-sm-yoon/starred{/owner}{/repo}
https://api.github.com/users/bv-sm-yoon/subscriptions
User
https://api.github.com/users/bv-sm-yoon
public
MEMBER
No problem!

2022-03-08T15:18:56Z
https://github.com/huggingface/datasets/issues/3831#issuecomment-1061889827
https://api.github.com/repos/huggingface/datasets/issues/3831
IC_kwDODunzps4_Sycj
https://api.github.com/repos/huggingface/datasets/issues/comments/1061889827/reactions
2022-03-08T15:18:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1061889827
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
## Describe the bug
when converting a dataset to tf_dataset by using to_tf_dataset with shuffle true, the remainder is not converted to one batch

## Steps to reproduce the bug
this is the sample code below
https://colab.research.google.com/drive/1_oRXWsR38ElO1EYF9ayFoCU7Ou1AAej4?usp=sharing


## Expected results
regardless of shuffle is true or not, 67 rows dataset should be 5 batches when batch size is 16.

## Actual results
4 batches

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1
