Allow loading dataset in-memory
MEMBER
I am wondering how to test their difference...
2021-01-29T15:18:05Z
https://github.com/huggingface/datasets/pull/1792#issuecomment-769866757
https://api.github.com/repos/huggingface/datasets/issues/1792
MDEyOklzc3VlQ29tbWVudDc2OTg2Njc1Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/769866757/reactions
2021-01-29T15:18:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/769866757
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
> ring how to test their difference...

Hmm I don't think pyarrow exposes an API to check if a Table comes from a file that is memory-mapped. In particular since all the buffer/memory logic is in the C++ part of pyarrow.

Otherwise we can still check the difference of RAM used when loading a big chunk of data.
2021-01-29T17:00:19Z
https://github.com/huggingface/datasets/pull/1792#issuecomment-769926140
https://api.github.com/repos/huggingface/datasets/issues/1792
MDEyOklzc3VlQ29tbWVudDc2OTkyNjE0MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/769926140/reactions
2021-01-29T17:00:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/769926140
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> Hmm I don't think pyarrow exposes an API to check if a Table comes from a file that is memory-mapped. In particular since all the buffer/memory logic is in the C++ part of pyarrow.
> 
> Otherwise we can still check the difference of RAM used when loading a big chunk of data.

@lhoestq I think I found a way: `pa.total_allocated_bytes()` :smirk:
2021-01-30T19:53:40Z
https://github.com/huggingface/datasets/pull/1792#issuecomment-770271543
https://api.github.com/repos/huggingface/datasets/issues/1792
MDEyOklzc3VlQ29tbWVudDc3MDI3MTU0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/770271543/reactions
2021-01-30T19:54:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/770271543
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Allow loading datasets either from:
- memory-mapped file (current implementation)
- from file descriptor, copying data to physical memory

Close #708