Faster map w/ input_columns & faster slicing w/ Iterable keys
CONTRIBUTOR
@lhoestq Just fixed the code style issues— I think it should be good to merge now :)
2021-04-23T17:56:35Z
https://github.com/huggingface/datasets/pull/2246#issuecomment-825823468
https://api.github.com/repos/huggingface/datasets/issues/2246
MDEyOklzc3VlQ29tbWVudDgyNTgyMzQ2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/825823468/reactions
2021-04-23T17:56:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/825823468
https://avatars.githubusercontent.com/u/39116809?v=4
https://api.github.com/users/norabelrose/events{/privacy}
https://api.github.com/users/norabelrose/followers
https://api.github.com/users/norabelrose/following{/other_user}
https://api.github.com/users/norabelrose/gists{/gist_id}
https://github.com/norabelrose
norabelrose
MDQ6VXNlcjM5MTE2ODA5
https://api.github.com/users/norabelrose/orgs
https://api.github.com/users/norabelrose/received_events
https://api.github.com/users/norabelrose/repos
https://api.github.com/users/norabelrose/starred{/owner}{/repo}
https://api.github.com/users/norabelrose/subscriptions
User
https://api.github.com/users/norabelrose
public
@lhoestq Fixes #2193 

- `map` now uses `with_format` to only load needed columns in memory when `input_columns` is set
- Slicing datasets with Iterables of indices now uses a new `Table.fast_gather` method, implemented with `np.searchsorted`, to find the appropriate batch indices all at once. `pa.concat_tables` is no longer used for this; we just call `pa.Table.from_batches` with a list of all the batch slices.

Together these changes have sped up batched `map()` calls over subsets of columns quite considerably in my initial testing.