Filter on dataset too much slowww
NONE
When I use the filter on the arrow table directly, it works like butter. But I can't find a way to update the table in `Dataset` object.

```
ds_table = dataset.data.filter(mask=dataset['flag'])
```
2021-01-30T04:13:39Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-770153899
https://api.github.com/repos/huggingface/datasets/issues/1796
MDEyOklzc3VlQ29tbWVudDc3MDE1Mzg5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/770153899/reactions
2021-01-30T04:13:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/770153899
https://avatars.githubusercontent.com/u/20911334?v=4
https://api.github.com/users/ayubSubhaniya/events{/privacy}
https://api.github.com/users/ayubSubhaniya/followers
https://api.github.com/users/ayubSubhaniya/following{/other_user}
https://api.github.com/users/ayubSubhaniya/gists{/gist_id}
https://github.com/ayubSubhaniya
ayubSubhaniya
MDQ6VXNlcjIwOTExMzM0
https://api.github.com/users/ayubSubhaniya/orgs
https://api.github.com/users/ayubSubhaniya/received_events
https://api.github.com/users/ayubSubhaniya/repos
https://api.github.com/users/ayubSubhaniya/starred{/owner}{/repo}
https://api.github.com/users/ayubSubhaniya/subscriptions
User
https://api.github.com/users/ayubSubhaniya
public
NONE
@thomwolf @lhoestq can you guys please take a look and recommend some solution.
2021-01-30T04:14:42Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-770153980
https://api.github.com/repos/huggingface/datasets/issues/1796
MDEyOklzc3VlQ29tbWVudDc3MDE1Mzk4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/770153980/reactions
2021-01-30T04:14:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/770153980
https://avatars.githubusercontent.com/u/20911334?v=4
https://api.github.com/users/ayubSubhaniya/events{/privacy}
https://api.github.com/users/ayubSubhaniya/followers
https://api.github.com/users/ayubSubhaniya/following{/other_user}
https://api.github.com/users/ayubSubhaniya/gists{/gist_id}
https://github.com/ayubSubhaniya
ayubSubhaniya
MDQ6VXNlcjIwOTExMzM0
https://api.github.com/users/ayubSubhaniya/orgs
https://api.github.com/users/ayubSubhaniya/received_events
https://api.github.com/users/ayubSubhaniya/repos
https://api.github.com/users/ayubSubhaniya/starred{/owner}{/repo}
https://api.github.com/users/ayubSubhaniya/subscriptions
User
https://api.github.com/users/ayubSubhaniya
public
MEMBER
Hi ! Currently the filter method reads the dataset batch by batch to write a new, filtered, arrow file on disk. Therefore all the reading + writing can take some time.
Using a mask directly on the arrow table doesn't do any read or write operation therefore it's way quicker.

Replacing the old table by the new one should do the job:
```python
dataset._data = dataset._data.filter(...)
```

Note: this is a **workaround** and in general users shouldn't have to do that. In particular if you did some `shuffle` or `select` before that then it would not work correctly since the indices mapping (index from `__getitem__` -> index in the table) would not be valid anymore. But if you haven't done any `shuffle`, `select`, `shard`, `train_test_split` etc. then it should work.

Ideally it would be awesome to update the filter function to allow masking this way !
If you would like to give it a shot I will be happy to help :) 
2021-02-01T09:35:22Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-770716800
https://api.github.com/repos/huggingface/datasets/issues/1796
MDEyOklzc3VlQ29tbWVudDc3MDcxNjgwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/770716800/reactions
2021-02-01T09:35:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/770716800
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Yes, would be happy to contribute. Thanks
2021-02-03T15:28:33Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-772594290
https://api.github.com/repos/huggingface/datasets/issues/1796
MDEyOklzc3VlQ29tbWVudDc3MjU5NDI5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/772594290/reactions
2021-02-03T15:28:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/772594290
https://avatars.githubusercontent.com/u/20911334?v=4
https://api.github.com/users/ayubSubhaniya/events{/privacy}
https://api.github.com/users/ayubSubhaniya/followers
https://api.github.com/users/ayubSubhaniya/following{/other_user}
https://api.github.com/users/ayubSubhaniya/gists{/gist_id}
https://github.com/ayubSubhaniya
ayubSubhaniya
MDQ6VXNlcjIwOTExMzM0
https://api.github.com/users/ayubSubhaniya/orgs
https://api.github.com/users/ayubSubhaniya/received_events
https://api.github.com/users/ayubSubhaniya/repos
https://api.github.com/users/ayubSubhaniya/starred{/owner}{/repo}
https://api.github.com/users/ayubSubhaniya/subscriptions
User
https://api.github.com/users/ayubSubhaniya
public
CONTRIBUTOR
Hi @lhoestq @ayubSubhaniya,

If there's no progress on this one, can I try working on it?

Thanks,
Gunjan
2021-02-18T14:03:17Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-781365027
https://api.github.com/repos/huggingface/datasets/issues/1796
MDEyOklzc3VlQ29tbWVudDc4MTM2NTAyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/781365027/reactions
2021-02-18T14:03:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/781365027
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
MEMBER
Sure @gchhablani feel free to start working on it, this would be very appreciated :)
This feature is would be really awesome, especially since arrow allows to mask really quickly and without having to rewrite the dataset on disk
2021-02-18T17:09:24Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-781496298
https://api.github.com/repos/huggingface/datasets/issues/1796
MDEyOklzc3VlQ29tbWVudDc4MTQ5NjI5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/781496298/reactions
2021-02-18T17:09:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/781496298
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq, any updates on this issue? The `filter` method is still veryyy slow 😕 
2024-01-16T19:41:23Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-1894397498
https://api.github.com/repos/huggingface/datasets/issues/1796
IC_kwDODunzps5w6jY6
https://api.github.com/repos/huggingface/datasets/issues/comments/1894397498/reactions
2024-01-16T19:41:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1894397498
https://avatars.githubusercontent.com/u/56069179?v=4
https://api.github.com/users/susnato/events{/privacy}
https://api.github.com/users/susnato/followers
https://api.github.com/users/susnato/following{/other_user}
https://api.github.com/users/susnato/gists{/gist_id}
https://github.com/susnato
susnato
MDQ6VXNlcjU2MDY5MTc5
https://api.github.com/users/susnato/orgs
https://api.github.com/users/susnato/received_events
https://api.github.com/users/susnato/repos
https://api.github.com/users/susnato/starred{/owner}{/repo}
https://api.github.com/users/susnato/subscriptions
User
https://api.github.com/users/susnato
public
MEMBER
No update so far, we haven't worked on this yet :/

Though PyArrow is much more stable than 3 years ago so it would be a good time to dive into this
2024-01-18T13:06:00Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-1898445864
https://api.github.com/repos/huggingface/datasets/issues/1796
IC_kwDODunzps5xJ_wo
https://api.github.com/repos/huggingface/datasets/issues/comments/1898445864/reactions
2024-01-18T13:06:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1898445864
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq, thanks a lot for the update! 

I would like to work on this(if possible). Could you please give me some steps regarding how should I approach this? Also any references would be great! 
2024-01-18T18:22:07Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-1898992678
https://api.github.com/repos/huggingface/datasets/issues/1796
IC_kwDODunzps5xMFQm
https://api.github.com/repos/huggingface/datasets/issues/comments/1898992678/reactions
2024-01-18T18:22:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1898992678
https://avatars.githubusercontent.com/u/56069179?v=4
https://api.github.com/users/susnato/events{/privacy}
https://api.github.com/users/susnato/followers
https://api.github.com/users/susnato/following{/other_user}
https://api.github.com/users/susnato/gists{/gist_id}
https://github.com/susnato
susnato
MDQ6VXNlcjU2MDY5MTc5
https://api.github.com/users/susnato/orgs
https://api.github.com/users/susnato/received_events
https://api.github.com/users/susnato/repos
https://api.github.com/users/susnato/starred{/owner}{/repo}
https://api.github.com/users/susnato/subscriptions
User
https://api.github.com/users/susnato
public
MEMBER
I just played a bit with it to make sure using `table.filter()` is fine, but actually it seems to create a new table **in memory** :/
This is an issue since it can quickly fill the RAM, and `datasets`'s role is to make sure you can load bigger-than-memory datasets. Therefore I don't think it's a good idea in the end to use `table.filter()`

Anyway I just ran OP's code an it runs in 20ms now on my side thanks to the I/O optimizations we did.

Another way to speed up `filter` is to add support pyarrow expressions though, using e.g. arrow formatting + dataset.filter (runs in 10ms on my side):

```python
import pyarrow.dataset as pds
import pyarrow.compute as pc

expr = pc.field("flag") == True  # or any condition

filtered = dataset.with_format("arrow").filter(
    lambda t: pds.dataset(t).to_table(columns={"mask": expr})[0].to_numpy(),
    batched=True,
).with_format(None)
```

or simply with numpy:

```python
filtered = dataset.with_format("np").filter(
    lambda flags_array: flags_array == True,  # or any condition
    input_columns=["flag"]
    batched=True,
).with_format(None)
2024-01-19T13:25:20Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-1900423162
https://api.github.com/repos/huggingface/datasets/issues/1796
IC_kwDODunzps5xRif6
https://api.github.com/repos/huggingface/datasets/issues/comments/1900423162/reactions
2025-05-15T13:18:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1900423162
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I can confirm that it's still extremely slow to use the `filter` function, and even trying workarounds like iterating on the dataset to create a mask for the `select` function is very slow (10-20 it/s) on a very small dataset with 5 fields.
2025-05-15T07:51:32Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-2882899759
https://api.github.com/repos/huggingface/datasets/issues/1796
IC_kwDODunzps6r1Y8v
https://api.github.com/repos/huggingface/datasets/issues/comments/2882899759/reactions
2025-05-15T07:51:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2882899759
https://avatars.githubusercontent.com/u/5941942?v=4
https://api.github.com/users/cpcdoy/events{/privacy}
https://api.github.com/users/cpcdoy/followers
https://api.github.com/users/cpcdoy/following{/other_user}
https://api.github.com/users/cpcdoy/gists{/gist_id}
https://github.com/cpcdoy
cpcdoy
MDQ6VXNlcjU5NDE5NDI=
https://api.github.com/users/cpcdoy/orgs
https://api.github.com/users/cpcdoy/received_events
https://api.github.com/users/cpcdoy/repos
https://api.github.com/users/cpcdoy/starred{/owner}{/repo}
https://api.github.com/users/cpcdoy/subscriptions
User
https://api.github.com/users/cpcdoy
public
MEMBER
You can iterate faster on the dataset if you format it with zero-copy formats like `with_format("arrow")`, `with_format("numpy")` or `with_format("polars")`. This is useful to use arrow/numpy/polars functions in map() or filter() and accelerate your data pipeline
2025-05-15T13:19:53Z
https://github.com/huggingface/datasets/issues/1796#issuecomment-2883790057
https://api.github.com/repos/huggingface/datasets/issues/1796
IC_kwDODunzps6r4yTp
https://api.github.com/repos/huggingface/datasets/issues/comments/2883790057/reactions
2025-05-15T13:19:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2883790057
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I have a dataset with 50M rows.
For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 12mins. I used `map()` with batch size 1024 and multi-process with 96 processes.

When I applied the `filter()` function it is taking too much time. I need to filter sequences based on a boolean column.
Below are the variants I tried.
1. filter() with batch size 1024, single process (takes roughly 3 hr)
2. filter() with batch size 1024, 96 processes (takes 5-6 hrs ¯\\\_(ツ)\_/¯)
3. filter() with loading all data in memory, only a single boolean column (never ends).

Can someone please help?

Below is a sample code for small dataset.

```
from datasets import load_dataset
dataset = load_dataset('glue', 'mrpc', split='train')
dataset = dataset.map(lambda x: {'flag': random.randint(0,1)==1})

def _amplify(data):
        return data

dataset = dataset.filter(_amplify, batch_size=1024, keep_in_memory=False, input_columns=['flag'])
```
