Bug with filtered indices
MEMBER
The issue is here:
https://github.com/huggingface/datasets/blob/3ad9644b9a2e4558dd1d0f1e43c67658674e6228/src/datasets/arrow_dataset.py#L2964
2022-10-14T10:39:27Z
https://github.com/huggingface/datasets/issues/5112#issuecomment-1278831370
https://api.github.com/repos/huggingface/datasets/issues/5112
IC_kwDODunzps5MOWsK
https://api.github.com/repos/huggingface/datasets/issues/comments/1278831370/reactions
2022-10-14T10:39:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1278831370
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
@PartiallyTyped, @Muennighoff: the issue is fixed.

We are planning to make a patch release today.
2022-10-14T13:44:31Z
https://github.com/huggingface/datasets/issues/5112#issuecomment-1279030553
https://api.github.com/repos/huggingface/datasets/issues/5112
IC_kwDODunzps5MPHUZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1279030553/reactions
2022-10-14T13:44:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1279030553
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Thanks a lot for the swift response! For a brief moment yesterday I thought I had gone insane 🤣On 14 Oct 2022, at 15:44, Albert Villanova del Moral ***@***.***> wrote:﻿
@PartiallyTyped, @Muennighoff: the issue is fixed.
We are planning to make a patch release today.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
2022-10-14T13:55:02Z
https://github.com/huggingface/datasets/issues/5112#issuecomment-1279043923
https://api.github.com/repos/huggingface/datasets/issues/5112
IC_kwDODunzps5MPKlT
https://api.github.com/repos/huggingface/datasets/issues/comments/1279043923/reactions
2022-10-14T13:55:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1279043923
https://avatars.githubusercontent.com/u/52372765?v=4
https://api.github.com/users/PartiallyUntyped/events{/privacy}
https://api.github.com/users/PartiallyUntyped/followers
https://api.github.com/users/PartiallyUntyped/following{/other_user}
https://api.github.com/users/PartiallyUntyped/gists{/gist_id}
https://github.com/PartiallyUntyped
PartiallyUntyped
MDQ6VXNlcjUyMzcyNzY1
https://api.github.com/users/PartiallyUntyped/orgs
https://api.github.com/users/PartiallyUntyped/received_events
https://api.github.com/users/PartiallyUntyped/repos
https://api.github.com/users/PartiallyUntyped/starred{/owner}{/repo}
https://api.github.com/users/PartiallyUntyped/subscriptions
User
https://api.github.com/users/PartiallyUntyped
public
## Describe the bug
As reported by @PartiallyTyped (and by @Muennighoff):
- https://github.com/huggingface/datasets/issues/5111#issuecomment-1278652524

There is an issue with the indices of a filtered dataset.

## Steps to reproduce the bug
```python
ds = Dataset.from_dict({"num": [0, 1, 2, 3]})
ds = ds.filter(lambda num: num % 2 == 0, input_columns="num", batch_size=2)
assert all(item["num"] % 2 == 0 for item in ds)
```

## Expected results
The indices of the filtered dataset should correspond to the examples with "language" equals to "english".

## Actual results
Indices to items with other languages are included in the filtered dataset indices

## Preliminar investigation
It seems a bug introduced by:
- #5030
