Fix cast memory usage by using map on subtables
MEMBER
I addressed your comments about the docstrings and the output validation :)
2021-04-12T17:51:04Z
https://github.com/huggingface/datasets/pull/2178#issuecomment-818006408
https://api.github.com/repos/huggingface/datasets/issues/2178
MDEyOklzc3VlQ29tbWVudDgxODAwNjQwOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/818006408/reactions
2021-04-12T17:51:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/818006408
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I updated the bleurt mocking method and bleurt test is passing now.
I also ran the slow tests and they are passing for bleurt.
2021-04-13T09:12:37Z
https://github.com/huggingface/datasets/pull/2178#issuecomment-818582660
https://api.github.com/repos/huggingface/datasets/issues/2178
MDEyOklzc3VlQ29tbWVudDgxODU4MjY2MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/818582660/reactions
2021-04-13T09:12:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/818582660
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks @lhoestq and @albertvillanova !
2021-04-13T11:39:57Z
https://github.com/huggingface/datasets/pull/2178#issuecomment-818668698
https://api.github.com/repos/huggingface/datasets/issues/2178
MDEyOklzc3VlQ29tbWVudDgxODY2ODY5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/818668698/reactions
2021-04-13T11:39:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/818668698
https://avatars.githubusercontent.com/u/33657802?v=4
https://api.github.com/users/SBrandeis/events{/privacy}
https://api.github.com/users/SBrandeis/followers
https://api.github.com/users/SBrandeis/following{/other_user}
https://api.github.com/users/SBrandeis/gists{/gist_id}
https://github.com/SBrandeis
SBrandeis
MDQ6VXNlcjMzNjU3ODAy
https://api.github.com/users/SBrandeis/orgs
https://api.github.com/users/SBrandeis/received_events
https://api.github.com/users/SBrandeis/repos
https://api.github.com/users/SBrandeis/starred{/owner}{/repo}
https://api.github.com/users/SBrandeis/subscriptions
User
https://api.github.com/users/SBrandeis
public
The `cast` operation on a pyarrow Table may create new arrays in memory.
This is an issue since users expect memory mapped datasets to not fill up the RAM.

To fix that I used `map` to write a new arrow file on disk when cast is used.
To make things more convenient I introduced the `arrow` formatting of a dataset, to make it return pyarrow tables instead of python dicts. This way one can use pyarrow transforms directly when using `map`.

edit: we'll use the same mechanism for `filter`