Make numpy arrow extractor faster
MEMBER
Looks like we have a nice speed up in some benchmarks. For example:
- `read_formatted numpy 5000`:  4.584777 sec -> 0.487113 sec
- `read_formatted torch 5000`:  4.565676 sec -> 1.289514 sec
2021-06-15T10:54:19Z
https://github.com/huggingface/datasets/pull/2505#issuecomment-861399194
https://api.github.com/repos/huggingface/datasets/issues/2505
MDEyOklzc3VlQ29tbWVudDg2MTM5OTE5NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/861399194/reactions
2021-06-15T10:54:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/861399194
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Can we convert this draft to PR @lhoestq ?
2021-06-18T11:43:11Z
https://github.com/huggingface/datasets/pull/2505#issuecomment-863977560
https://api.github.com/repos/huggingface/datasets/issues/2505
MDEyOklzc3VlQ29tbWVudDg2Mzk3NzU2MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/863977560/reactions
2021-06-18T11:43:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/863977560
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
MEMBER
Ready for review ! cc @vblagoje
2021-06-21T16:12:49Z
https://github.com/huggingface/datasets/pull/2505#issuecomment-865165544
https://api.github.com/repos/huggingface/datasets/issues/2505
MDEyOklzc3VlQ29tbWVudDg2NTE2NTU0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/865165544/reactions
2021-06-21T16:12:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/865165544
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq I tried the branch and it works for me. Although performance trace now shows a speedup, the overall pre-training speed up is minimal. But that's on my plate to explore further. 
2021-06-22T09:51:23Z
https://github.com/huggingface/datasets/pull/2505#issuecomment-865838880
https://api.github.com/repos/huggingface/datasets/issues/2505
MDEyOklzc3VlQ29tbWVudDg2NTgzODg4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/865838880/reactions
2021-06-22T09:51:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/865838880
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
MEMBER
Thanks for investigating @vblagoje 

@albertvillanova , do you have any comments on this PR ? Otherwise I think we can merge it
2021-06-22T10:19:01Z
https://github.com/huggingface/datasets/pull/2505#issuecomment-865858348
https://api.github.com/repos/huggingface/datasets/issues/2505
MDEyOklzc3VlQ29tbWVudDg2NTg1ODM0OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/865858348/reactions
2021-06-22T10:19:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/865858348
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I changed the NumpyArrowExtractor to call directly to_numpy and see if it can lead to speed-ups as discussed in https://github.com/huggingface/datasets/issues/2498

This could make the numpy/torch/tf/jax formatting faster