Enable Fast Filtering using Arrow Dataset
MEMBER
Hi @gchhablani :)
Thanks for proposing your help !

I'll be doing a refactor of some parts related to filtering in the scope of https://github.com/huggingface/datasets/issues/1877
So I would first wait for this refactor to be done before working on the filtering. In particular because I plan to make things simpler to manipulate.

Your feedback on this refactor would also be appreciated since it also aims at making the core code more accessible (basically my goal is that no one's ever "having troubles getting started" ^^)

This will be available in a few days, I will be able to give you more details at that time if you don't mind waiting a bit !
2021-02-26T10:51:39Z
https://github.com/huggingface/datasets/issues/1949#issuecomment-786571223
https://api.github.com/repos/huggingface/datasets/issues/1949
MDEyOklzc3VlQ29tbWVudDc4NjU3MTIyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/786571223/reactions
2021-02-26T10:51:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/786571223
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Sure! I don't mind waiting. I'll check the refactor and try to understand what you're trying to do :)
2021-02-26T19:17:35Z
https://github.com/huggingface/datasets/issues/1949#issuecomment-786843151
https://api.github.com/repos/huggingface/datasets/issues/1949
MDEyOklzc3VlQ29tbWVudDc4Njg0MzE1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/786843151/reactions
2021-02-26T19:18:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/786843151
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
Hi @lhoestq,

As mentioned in Issue #1796, I would love to work on enabling fast filtering/mapping. Can you please share the expectations? It would be great if you could point me to the relevant methods/files involved. Or the docs or maybe an overview of `arrow_dataset.py`. I only ask this because I am having trouble getting started ;-;

Any help would be appreciated.

Thanks,
Gunjan