combination of shuffle and filter results in a bug
CONTRIBUTOR
I cannot reproduce this on master and pyarrow==4.0.1.

2021-11-01T15:30:31Z
https://github.com/huggingface/datasets/issues/3190#issuecomment-956334601
https://api.github.com/repos/huggingface/datasets/issues/3190
IC_kwDODunzps45AIIJ
https://api.github.com/repos/huggingface/datasets/issues/comments/956334601/reactions
2021-11-01T15:30:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/956334601
https://avatars.githubusercontent.com/u/2779410?v=4
https://api.github.com/users/BramVanroy/events{/privacy}
https://api.github.com/users/BramVanroy/followers
https://api.github.com/users/BramVanroy/following{/other_user}
https://api.github.com/users/BramVanroy/gists{/gist_id}
https://github.com/BramVanroy
BramVanroy
MDQ6VXNlcjI3Nzk0MTA=
https://api.github.com/users/BramVanroy/orgs
https://api.github.com/users/BramVanroy/received_events
https://api.github.com/users/BramVanroy/repos
https://api.github.com/users/BramVanroy/starred{/owner}{/repo}
https://api.github.com/users/BramVanroy/subscriptions
User
https://api.github.com/users/BramVanroy
public
MEMBER
Hi ! There was a regression in `datasets` 1.12 that introduced this bug. It has been fixed in #3019 in 1.13

Can you try to update `datasets` and try again ?
2021-11-02T09:57:45Z
https://github.com/huggingface/datasets/issues/3190#issuecomment-957283469
https://api.github.com/repos/huggingface/datasets/issues/3190
IC_kwDODunzps45DvyN
https://api.github.com/repos/huggingface/datasets/issues/comments/957283469/reactions
2021-11-02T09:57:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/957283469
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks a lot, fixes with 1.13
2021-11-02T10:50:49Z
https://github.com/huggingface/datasets/issues/3190#issuecomment-957327035
https://api.github.com/repos/huggingface/datasets/issues/3190
IC_kwDODunzps45D6a7
https://api.github.com/repos/huggingface/datasets/issues/comments/957327035/reactions
2021-11-02T10:50:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/957327035
https://avatars.githubusercontent.com/u/6278280?v=4
https://api.github.com/users/rabeehk/events{/privacy}
https://api.github.com/users/rabeehk/followers
https://api.github.com/users/rabeehk/following{/other_user}
https://api.github.com/users/rabeehk/gists{/gist_id}
https://github.com/rabeehk
rabeehk
MDQ6VXNlcjYyNzgyODA=
https://api.github.com/users/rabeehk/orgs
https://api.github.com/users/rabeehk/received_events
https://api.github.com/users/rabeehk/repos
https://api.github.com/users/rabeehk/starred{/owner}{/repo}
https://api.github.com/users/rabeehk/subscriptions
User
https://api.github.com/users/rabeehk
public
## Describe the bug
Hi,
I would like to shuffle a dataset, then filter it based on each existing label. however, the combination of `filter`, `shuffle` seems to results in a bug. In the minimal example below, as you see in the filtered results, the filtered labels are not unique, meaning filter has not worked. Any suggestions as a temporary fix is appreciated @lhoestq.

Thanks.
 Best regards
Rabeeh 

## Steps to reproduce the bug
```python
import numpy as np
import datasets 

datasets = datasets.load_dataset('super_glue', 'rte', script_version="master")
shuffled_data = datasets["train"].shuffle(seed=42)
for label in range(2):
    print("label ", label)
    data = shuffled_data.filter(lambda example: int(example['label']) == label)
    print("length ", len(data), np.unique(data['label']))
```

## Expected results
Filtering per label, should only return the data with that specific label.

## Actual results
As you can see, filtered data per label, has still two labels of [0, 1]
```
label  0
length  1249 [0 1]
label  1
length  1241 [0 1]
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1 
- Platform: linux 
- Python version: 3.7.11 
- PyArrow version: 5.0.0 
