More efficient nested features encoding
CONTRIBUTOR
@lhoestq @albertvillanova @mariosasko
Can you please check this out?
2021-10-29T14:07:11Z
https://github.com/huggingface/datasets/pull/3124#issuecomment-954774268
https://api.github.com/repos/huggingface/datasets/issues/3124
IC_kwDODunzps446LL8
https://api.github.com/repos/huggingface/datasets/issues/comments/954774268/reactions
2021-10-29T14:07:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/954774268
https://avatars.githubusercontent.com/u/13485709?v=4
https://api.github.com/users/eladsegal/events{/privacy}
https://api.github.com/users/eladsegal/followers
https://api.github.com/users/eladsegal/following{/other_user}
https://api.github.com/users/eladsegal/gists{/gist_id}
https://github.com/eladsegal
eladsegal
MDQ6VXNlcjEzNDg1NzA5
https://api.github.com/users/eladsegal/orgs
https://api.github.com/users/eladsegal/received_events
https://api.github.com/users/eladsegal/repos
https://api.github.com/users/eladsegal/starred{/owner}{/repo}
https://api.github.com/users/eladsegal/subscriptions
User
https://api.github.com/users/eladsegal
public
CONTRIBUTOR
Thanks, done!
2021-10-29T16:14:45Z
https://github.com/huggingface/datasets/pull/3124#issuecomment-954871978
https://api.github.com/repos/huggingface/datasets/issues/3124
IC_kwDODunzps446jCq
https://api.github.com/repos/huggingface/datasets/issues/comments/954871978/reactions
2021-10-29T16:14:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/954871978
https://avatars.githubusercontent.com/u/13485709?v=4
https://api.github.com/users/eladsegal/events{/privacy}
https://api.github.com/users/eladsegal/followers
https://api.github.com/users/eladsegal/following{/other_user}
https://api.github.com/users/eladsegal/gists{/gist_id}
https://github.com/eladsegal
eladsegal
MDQ6VXNlcjEzNDg1NzA5
https://api.github.com/users/eladsegal/orgs
https://api.github.com/users/eladsegal/received_events
https://api.github.com/users/eladsegal/repos
https://api.github.com/users/eladsegal/starred{/owner}{/repo}
https://api.github.com/users/eladsegal/subscriptions
User
https://api.github.com/users/eladsegal
public
Nested encoding of features wastes a lot of time on operations which are effectively doing nothing when lists are used.
For example, if in the input we have a list of integers, `encoded_nested_example` will iterate over it and apply `encoded_nested_example` on every element even though it just return the int as is.

A similar issue is handled at an earlier stage when casting pytorch/tensorflow/pandas objects to python lists/numpy arrays:
https://github.com/huggingface/datasets/blob/c98c23c4260edadab00f997d1a5d66b7f2e93ce9/src/datasets/features/features.py#L149-L156
https://github.com/huggingface/datasets/blob/c98c23c4260edadab00f997d1a5d66b7f2e93ce9/src/datasets/features/features.py#L212-L228

In this pull request I suggest to use the same approach in `encoded_nested_example`.
In my setup there was a major speedup with this change: loading the data was at least x4 faster. 