Performance regression in `dataset.filter`
MEMBER
Thanks for reporting, I'll fix the regression today
2025-02-17T14:20:32Z
https://github.com/huggingface/datasets/issues/7404#issuecomment-2663271077
https://api.github.com/repos/huggingface/datasets/issues/7404
IC_kwDODunzps6evkql
https://api.github.com/repos/huggingface/datasets/issues/comments/2663271077/reactions
2025-02-17T14:20:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2663271077
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I just released `datasets` 3.3.1 with a fix, let me know if it's good now :)
2025-02-17T14:55:29Z
https://github.com/huggingface/datasets/issues/7404#issuecomment-2663361449
https://api.github.com/repos/huggingface/datasets/issues/7404
IC_kwDODunzps6ev6up
https://api.github.com/repos/huggingface/datasets/issues/comments/2663361449/reactions
2025-02-17T14:55:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2663361449
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq it fixed the issue.

This was (very) fast, thank you very much!
2025-02-17T17:46:05Z
https://github.com/huggingface/datasets/issues/7404#issuecomment-2663778425
https://api.github.com/repos/huggingface/datasets/issues/7404
IC_kwDODunzps6exgh5
https://api.github.com/repos/huggingface/datasets/issues/comments/2663778425/reactions
2025-02-17T17:46:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2663778425
https://avatars.githubusercontent.com/u/82200?v=4
https://api.github.com/users/ttim/events{/privacy}
https://api.github.com/users/ttim/followers
https://api.github.com/users/ttim/following{/other_user}
https://api.github.com/users/ttim/gists{/gist_id}
https://github.com/ttim
ttim
MDQ6VXNlcjgyMjAw
https://api.github.com/users/ttim/orgs
https://api.github.com/users/ttim/received_events
https://api.github.com/users/ttim/repos
https://api.github.com/users/ttim/starred{/owner}{/repo}
https://api.github.com/users/ttim/subscriptions
User
https://api.github.com/users/ttim
public
### Describe the bug

We're filtering dataset of ~1M (small-ish) records. At some point in the code we do `dataset.filter`, before (including 3.2.0) it was taking couple of seconds, and now it takes 4 hours.

We use 16 threads/workers, and stack trace at them look as follows:
```
Traceback (most recent call last):
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/python/lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3511, in _map_single
    for i, batch in iter_outputs(shard_iterable):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3461, in iter_outputs
    yield i, apply_function(example, i, offset=offset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3390, in apply_function
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 6416, in get_indices_from_mask_function
    indices_array = indices_mapping.column(0).take(indices_array)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 1079, in pyarrow.lib.ChunkedArray.take
  File "/python/lib/python3.12/site-packages/pyarrow/compute.py", line 458, in take
    def take(data, indices, *, boundscheck=True, memory_pool=None):
```

### Steps to reproduce the bug

1. Save dataset of 1M records in arrow
2. Filter it with 16 threads
3. Watch it take too long

### Expected behavior

Filtering done fast

### Environment info

datasets 3.3.0, python 3.12