Add env variable for MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES
CONTRIBUTOR
Thank you for clarifying the precedence, @albertvillanova 

Isn't it typically the case where env vars have the highest precedence? 

In my understanding the point of env vars is to be able to override software w/o needing to touch the code. 

Please correct me if this is not so in the general case.
2021-05-25T16:49:46Z
https://github.com/huggingface/datasets/pull/2399#issuecomment-848044097
https://api.github.com/repos/huggingface/datasets/issues/2399
MDEyOklzc3VlQ29tbWVudDg0ODA0NDA5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/848044097/reactions
2021-05-25T16:50:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/848044097
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
MEMBER
Hi @stas00, 

Well, I'm not an expert on this topic, but the precedence hierarchy I have normally found is from higher to lower:
- command line parameters
- env vars
- config files
So yes, normally env vars have precedence over configuration files.

Anyway, for Datasets, there are no configuration files. The _in-memory_ config is set from default values or env vars (which have precedence over default values). But this is done at import.

However, once the library is imported, the user can modify the in-memory config, and this will have precedence over the rest of mechanisms (which take place only at import).
2021-05-25T17:12:17Z
https://github.com/huggingface/datasets/pull/2399#issuecomment-848059144
https://api.github.com/repos/huggingface/datasets/issues/2399
MDEyOklzc3VlQ29tbWVudDg0ODA1OTE0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/848059144/reactions
2021-05-25T17:12:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/848059144
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
In my limited experience env vars are typically above cmd line args, so that one can override scripts with cmd lines using env vars, but usually one then uses env vars inside cmd line args, so it's loud and clear.

For example specifying a specific gpu number on a command line will depend on `CUDA_VISIBLE_DEVICES` so gpu0 will be different if someone sets `CUDA_VISIBLE_DEVICES=2,3` vs  `CUDA_VISIBLE_DEVICES=0,1`.

> However, once the library is imported, the user can modify the in-memory config, and this will have precedence over the rest of mechanisms (which take place only at import).

And this is exactly the problem we are trying to solve here. For a good reason HF examples don't want to use `keep_in_memory=False`, and they may choose to now set `datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES` and which means we again can't override it via env var.


2021-05-25T21:12:28Z
https://github.com/huggingface/datasets/pull/2399#issuecomment-848267581
https://api.github.com/repos/huggingface/datasets/issues/2399
MDEyOklzc3VlQ29tbWVudDg0ODI2NzU4MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/848267581/reactions
2021-05-25T21:14:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/848267581
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
CONTRIBUTOR
oops, sorry, didn't think earlier - do we need to prefix this with `HF_DATASETS` or `HF_` like all the other env vars? or is it long enough already to be unique - it's just not telling the user in the config file what projet this variable is for...
2021-05-26T17:05:35Z
https://github.com/huggingface/datasets/pull/2399#issuecomment-848951221
https://api.github.com/repos/huggingface/datasets/issues/2399
MDEyOklzc3VlQ29tbWVudDg0ODk1MTIyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/848951221/reactions
2021-05-26T17:05:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/848951221
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
MEMBER
You're right, I just opened https://github.com/huggingface/datasets/pull/2409
2021-05-27T09:07:15Z
https://github.com/huggingface/datasets/pull/2399#issuecomment-849469665
https://api.github.com/repos/huggingface/datasets/issues/2399
MDEyOklzc3VlQ29tbWVudDg0OTQ2OTY2NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/849469665/reactions
2021-05-27T09:07:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/849469665
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Add env variable for `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES`.

This will allow to turn off default behavior: loading in memory (and not caching) small datasets.

Fix #2387.