map and filter not working properly in multiprocessing with the new release 2.6.0
NONE
Same bug exists with `num_proc=1` on colab. `3.7.14 (default, Sep  8 2022, 00:06:44) [GCC 7.5.0]` 
2022-10-13T20:10:14Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278125161
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MLqRp
https://api.github.com/repos/huggingface/datasets/issues/comments/1278125161/reactions
2022-10-13T20:10:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278125161
https://avatars.githubusercontent.com/u/52372765?v=4
https://api.github.com/users/PartiallyUntyped/events{/privacy}
https://api.github.com/users/PartiallyUntyped/followers
https://api.github.com/users/PartiallyUntyped/following{/other_user}
https://api.github.com/users/PartiallyUntyped/gists{/gist_id}
https://github.com/PartiallyUntyped
PartiallyUntyped
MDQ6VXNlcjUyMzcyNzY1
https://api.github.com/users/PartiallyUntyped/orgs
https://api.github.com/users/PartiallyUntyped/received_events
https://api.github.com/users/PartiallyUntyped/repos
https://api.github.com/users/PartiallyUntyped/starred{/owner}{/repo}
https://api.github.com/users/PartiallyUntyped/subscriptions
User
https://api.github.com/users/PartiallyUntyped
public
MEMBER
Thanks for reporting, @loubnabnl and for the additional information, @PartiallyTyped.

However, I'm not able to reproduce this issue, neither locally nor on Colab:
```
Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 10
})
Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 10
})
```
CC: @huggingface/datasets can anybody reproduce this?
2022-10-14T07:51:24Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278624114
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MNkFy
https://api.github.com/repos/huggingface/datasets/issues/comments/1278624114/reactions
2022-10-14T07:51:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278624114
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
This is the minimum reproducible example. I ran this on the premium instances of colab.

```
# !pip install datasets
import datasets
from datasets import load_dataset
ds = load_dataset("copenlu/answerable_tydiqa").filter("english".__eq__, input_columns="language")
assert all(map("english".__eq__, ds["train"]["language"]))
```

In my case, the number of samples is correct, however, the samples selected when indexing are wrong.

```python
DatasetDict({
    validation: Dataset({
        features: ['question_text', 'document_title', 'language', 'annotations', 'document_plaintext', 'document_url'],
        num_rows: 990
    })
    train: Dataset({
        features: ['question_text', 'document_title', 'language', 'annotations', 'document_plaintext', 'document_url'],
        num_rows: 7389
    })
})
```

The number of rows is indeed correct, and i have checked it with a version that works.
2022-10-14T08:15:22Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278652524
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MNrBs
https://api.github.com/repos/huggingface/datasets/issues/comments/1278652524/reactions
2022-10-14T08:17:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278652524
https://avatars.githubusercontent.com/u/52372765?v=4
https://api.github.com/users/PartiallyUntyped/events{/privacy}
https://api.github.com/users/PartiallyUntyped/followers
https://api.github.com/users/PartiallyUntyped/following{/other_user}
https://api.github.com/users/PartiallyUntyped/gists{/gist_id}
https://github.com/PartiallyUntyped
PartiallyUntyped
MDQ6VXNlcjUyMzcyNzY1
https://api.github.com/users/PartiallyUntyped/orgs
https://api.github.com/users/PartiallyUntyped/received_events
https://api.github.com/users/PartiallyUntyped/repos
https://api.github.com/users/PartiallyUntyped/starred{/owner}{/repo}
https://api.github.com/users/PartiallyUntyped/subscriptions
User
https://api.github.com/users/PartiallyUntyped
public
NONE
I can reproduce the issue on my mac too 
```
- `datasets` version: 2.6.0
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3
```
But not on Colab with python 3.7, maybe related to python version? (didn't manage to install python 3.9)
```
- `datasets` version: 2.6.0
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
```
2022-10-14T08:32:11Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278672386
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MNv4C
https://api.github.com/repos/huggingface/datasets/issues/comments/1278672386/reactions
2022-10-14T09:12:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278672386
https://avatars.githubusercontent.com/u/44069155?v=4
https://api.github.com/users/loubnabnl/events{/privacy}
https://api.github.com/users/loubnabnl/followers
https://api.github.com/users/loubnabnl/following{/other_user}
https://api.github.com/users/loubnabnl/gists{/gist_id}
https://github.com/loubnabnl
loubnabnl
MDQ6VXNlcjQ0MDY5MTU1
https://api.github.com/users/loubnabnl/orgs
https://api.github.com/users/loubnabnl/received_events
https://api.github.com/users/loubnabnl/repos
https://api.github.com/users/loubnabnl/starred{/owner}{/repo}
https://api.github.com/users/loubnabnl/subscriptions
User
https://api.github.com/users/loubnabnl
public
CONTRIBUTOR
I have the same issue, here's a simple notebook to reproduce: https://colab.research.google.com/drive/1Lvo9fg5DSpGUUgXW5JAutZ0bFsR-WV--?usp=sharing




2022-10-14T09:13:02Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278720996
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MN7vk
https://api.github.com/repos/huggingface/datasets/issues/comments/1278720996/reactions
2022-10-14T09:13:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278720996
https://avatars.githubusercontent.com/u/62820084?v=4
https://api.github.com/users/Muennighoff/events{/privacy}
https://api.github.com/users/Muennighoff/followers
https://api.github.com/users/Muennighoff/following{/other_user}
https://api.github.com/users/Muennighoff/gists{/gist_id}
https://github.com/Muennighoff
Muennighoff
MDQ6VXNlcjYyODIwMDg0
https://api.github.com/users/Muennighoff/orgs
https://api.github.com/users/Muennighoff/received_events
https://api.github.com/users/Muennighoff/repos
https://api.github.com/users/Muennighoff/starred{/owner}{/repo}
https://api.github.com/users/Muennighoff/subscriptions
User
https://api.github.com/users/Muennighoff
public
MEMBER
I think there are 2 different issues here:
- the one reported by @loubnabnl is related to multiprocessing in map and then filter; we should reproduce it first: I have tried with Python version 3.9.7 and I can't reproduce it either; maybe it is related to the version of PyArrow? To be checked.
- the issue reported by @PartiallyTyped is related just to "filter" (without multiprocessing) and I can reproduce it.
2022-10-14T10:26:03Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278812507
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MOSFb
https://api.github.com/repos/huggingface/datasets/issues/comments/1278812507/reactions
2022-10-14T10:26:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278812507
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Could you create another issue for the @PartiallyTyped one please ?

Regarding the OP issue, I also tried on colab or locally on py3.7 or py3.10 but didn't reproduce
2022-10-14T10:26:59Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278813602
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MOSWi
https://api.github.com/repos/huggingface/datasets/issues/comments/1278813602/reactions
2022-10-14T10:27:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278813602
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I have created another issue for the one reported by @PartiallyTyped: 
- #5112 
2022-10-14T10:36:39Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278828516
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MOV_k
https://api.github.com/repos/huggingface/datasets/issues/comments/1278828516/reactions
2022-10-14T10:36:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278828516
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I managed to reproduce your issue @loubnabnl on colab by upgrading pyarrow to 9.0.0 instead of 6.0.1
2022-10-14T10:57:21Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278849675
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MObKL
https://api.github.com/repos/huggingface/datasets/issues/comments/1278849675/reactions
2022-10-14T10:57:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278849675
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I managed to have a _super_ minimal reproducible example:
```python

from datasets import Dataset, concatenate_datasets

ds = concatenate_datasets([Dataset.from_dict({"a": [i]}) for i in range(10)])
ds2 = ds.map(lambda _: {}, batched=True)
assert list(ds2) == list(ds)
```
(filter uses a batched `map` under the hood)
2022-10-14T11:06:57Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278859284
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MOdgU
https://api.github.com/repos/huggingface/datasets/issues/comments/1278859284/reactions
2022-10-14T11:06:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278859284
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
>  the one reported by @loubnabnl is related to multiprocessing in map and then filter; we should reproduce it first: I have tried with Python version 3.9.7 and I can't reproduce it either; maybe it is related to the version of PyArrow? To be checked.

So finally it was related to PyArrow version! :+1: 
2022-10-14T11:43:46Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1278894687
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MOmJf
https://api.github.com/repos/huggingface/datasets/issues/comments/1278894687/reactions
2022-10-14T11:45:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278894687
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Doing a patch release asap :)
2022-10-14T15:09:09Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1279134500
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MPgsk
https://api.github.com/repos/huggingface/datasets/issues/comments/1279134500/reactions
2022-10-14T15:09:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1279134500
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Did the patch release yesterday, lmk if you still have issues
2022-10-15T07:31:36Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1279685017
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MRnGZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1279685017/reactions
2022-10-15T07:31:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1279685017
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
It works now, thanks!

2022-10-17T08:26:59Z
https://github.com/huggingface/datasets/issues/5111#issuecomment-1280479219
https://api.github.com/repos/huggingface/datasets/issues/5111
IC_kwDODunzps5MUo_z
https://api.github.com/repos/huggingface/datasets/issues/comments/1280479219/reactions
2022-10-17T08:26:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1280479219
https://avatars.githubusercontent.com/u/44069155?v=4
https://api.github.com/users/loubnabnl/events{/privacy}
https://api.github.com/users/loubnabnl/followers
https://api.github.com/users/loubnabnl/following{/other_user}
https://api.github.com/users/loubnabnl/gists{/gist_id}
https://github.com/loubnabnl
loubnabnl
MDQ6VXNlcjQ0MDY5MTU1
https://api.github.com/users/loubnabnl/orgs
https://api.github.com/users/loubnabnl/received_events
https://api.github.com/users/loubnabnl/repos
https://api.github.com/users/loubnabnl/starred{/owner}{/repo}
https://api.github.com/users/loubnabnl/subscriptions
User
https://api.github.com/users/loubnabnl
public
## Describe the bug
When mapping is used on a dataset with more than one process, there is a weird behavior when trying to use `filter` , it's like only the samples from one worker are retrieved, one needs to specify the same `num_proc` in filter for it to work properly. This doesn't happen with `datasets` version 2.5.2

In the code below the data is filtered differently when we increase `num_proc` used in `map` although the datsets before and after mapping have identical elements.

## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset

def preprocess(example):
    return example

ds = load_dataset("codeparrot/codeparrot-clean-valid", split="train").select([i for i in range(10)])
ds1 = ds.map(preprocess, num_proc=2)
ds2 = ds.map(preprocess)

# the datasets elements are the same
for i in range(len(ds1)):
    assert ds1[i]==ds2[i]

print(f'Target column before filtering {ds1["autogenerated"]}')
print(f'Target column before filtering {ds2["autogenerated"]}')
print(f"datasets version {datasets.__version__}")

ds_filtered_1 = ds1.filter(lambda x: not x["autogenerated"])
ds_filtered_2 = ds2.filter(lambda x: not x["autogenerated"])

# all elements in Target column are false so they should all be kept, but for ds2 only the first 5=num_samples/num_proc are kept
print(ds_filtered_1)
print(ds_filtered_2)
```
```
Target column before filtering [False, False, False, False, False, False, False, False, False, False]
Target column before filtering [False, False, False, False, False, False, False, False, False, False]

Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 5
})
Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 10
})
```
## Expected results
Increasing `num_proc` in mapping shouldn't alter filtering. With the previous version 2.5.2 this doesn't happen

## Actual results
Filtering doesn't work properly when we increase `num_proc` in mapping but not when calling `filter`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.6.0
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.2