Set default in-memory value depending on the dataset size
MEMBER
I ping @krandiash to keep him up to date.
2021-04-07T13:13:28Z
https://github.com/huggingface/datasets/pull/2182#issuecomment-814904068
https://api.github.com/repos/huggingface/datasets/issues/2182
MDEyOklzc3VlQ29tbWVudDgxNDkwNDA2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/814904068/reactions
2021-04-07T13:13:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814904068
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
TODO:
- [x] Add a section in the docs about this.
- ~Add a warning if someone tries to specify `cache_file_name=` in `map`, `filter` etc. on a dataset that is in memory, since the computation is not going to be cached in this case.~
2021-04-12T07:51:04Z
https://github.com/huggingface/datasets/pull/2182#issuecomment-817575851
https://api.github.com/repos/huggingface/datasets/issues/2182
MDEyOklzc3VlQ29tbWVudDgxNzU3NTg1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/817575851/reactions
2021-04-19T10:01:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/817575851
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
@lhoestq I have a question, regarding:
> Also maybe we should add a warning if someone tries to specify cache_file_name= in map, filter etc. on a dataset that is in memory, since the computation is not going to be cached in this case.

- It might be the case that the user has an in-memory dataset and might want to use `map` and cache it, by passing `cache_file_name=`
- This is indeed allowed by the library and works as expected: the dataset is cached.

Why adding a warning?
2021-04-13T09:14:06Z
https://github.com/huggingface/datasets/pull/2182#issuecomment-818583719
https://api.github.com/repos/huggingface/datasets/issues/2182
MDEyOklzc3VlQ29tbWVudDgxODU4MzcxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/818583719/reactions
2021-04-13T09:16:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/818583719
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Yes right, I meant if `load_from_cache_file` is set to True and `cache_file_name ` is None. my bad :p
2021-04-13T13:55:55Z
https://github.com/huggingface/datasets/pull/2182#issuecomment-818756271
https://api.github.com/repos/huggingface/datasets/issues/2182
MDEyOklzc3VlQ29tbWVudDgxODc1NjI3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/818756271/reactions
2021-04-13T13:56:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/818756271
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Set a default value for `in_memory` depending on the size of the dataset to be loaded.

Close #2179.

TODO:
- [x] Add a section in the docs about this.
- ~Add a warning if someone tries to specify `cache_file_name=` in `map`, `filter` etc. on a dataset that is in memory, since the computation is not going to be cached in this case.~