Optimize Dataset.filter to only compute the indices to keep
CONTRIBUTOR
Maybe worth updating the docs here as well?
2021-09-14T14:16:17Z
https://github.com/huggingface/datasets/pull/2836#issuecomment-919194988
https://api.github.com/repos/huggingface/datasets/issues/2836
IC_kwDODunzps42yc1s
https://api.github.com/repos/huggingface/datasets/issues/comments/919194988/reactions
2021-09-14T14:16:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/919194988
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
Yup, will do !
2021-09-14T14:51:52Z
https://github.com/huggingface/datasets/pull/2836#issuecomment-919227398
https://api.github.com/repos/huggingface/datasets/issues/2836
IC_kwDODunzps42ykwG
https://api.github.com/repos/huggingface/datasets/issues/comments/919227398/reactions
2021-09-14T14:51:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/919227398
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Optimize `Dataset.filter` to only compute the indices of the rows to keep, instead of creating a new Arrow table with the rows to keep. Creating a new table was an issue because it could take a lot of disk space.

This will be useful to process audio datasets for example cc @patrickvonplaten 