Added cache dirs to load and file_utils
MEMBER
hi ! the `hf_hub_download` cache_dir is a different cache directory than the one for `datasets`.

`hf_hub_download` uses the `huggingface_hub` cache which is located in by default in `~/.cache/huggingface/hub`, while `datasets` uses a different cache for Arrow files and map() results `~/.cache/huggingface/datasets`
2025-04-15T12:31:06Z
https://github.com/huggingface/datasets/pull/7499#issuecomment-2804895606
https://api.github.com/repos/huggingface/datasets/issues/7499
IC_kwDODunzps6nL092
https://api.github.com/repos/huggingface/datasets/issues/comments/2804895606/reactions
2025-04-15T12:31:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2804895606
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Is there a way to change the default cache directory for both of these on calling load_dataset? Currently, cache_dir makes dealing with where I want files to go a bit confusing as the documentation doesn't mention it only relocates.../datasets and not .../hub.
2025-04-15T12:52:40Z
https://github.com/huggingface/datasets/pull/7499#issuecomment-2804953408
https://api.github.com/repos/huggingface/datasets/issues/7499
IC_kwDODunzps6nMDFA
https://api.github.com/repos/huggingface/datasets/issues/comments/2804953408/reactions
2025-04-15T12:52:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2804953408
https://avatars.githubusercontent.com/u/43501738?v=4
https://api.github.com/users/gmongaras/events{/privacy}
https://api.github.com/users/gmongaras/followers
https://api.github.com/users/gmongaras/following{/other_user}
https://api.github.com/users/gmongaras/gists{/gist_id}
https://github.com/gmongaras
gmongaras
MDQ6VXNlcjQzNTAxNzM4
https://api.github.com/users/gmongaras/orgs
https://api.github.com/users/gmongaras/received_events
https://api.github.com/users/gmongaras/repos
https://api.github.com/users/gmongaras/starred{/owner}{/repo}
https://api.github.com/users/gmongaras/subscriptions
User
https://api.github.com/users/gmongaras
public
MEMBER
You can set `HF_HOME` which is the common parent directory for those two caches. Or individually `HF_DATASETS_CACHE` and `HF_HUB_CACHE`
2025-04-15T13:08:28Z
https://github.com/huggingface/datasets/pull/7499#issuecomment-2805003057
https://api.github.com/repos/huggingface/datasets/issues/7499
IC_kwDODunzps6nMPMx
https://api.github.com/repos/huggingface/datasets/issues/comments/2805003057/reactions
2025-04-15T13:08:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2805003057
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Got it. Can this be added to the documentation for load_dataset and related functions to avoid confusion with cache_dir?
2025-04-15T13:19:14Z
https://github.com/huggingface/datasets/pull/7499#issuecomment-2805034383
https://api.github.com/repos/huggingface/datasets/issues/7499
IC_kwDODunzps6nMW2P
https://api.github.com/repos/huggingface/datasets/issues/comments/2805034383/reactions
2025-04-15T13:19:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2805034383
https://avatars.githubusercontent.com/u/43501738?v=4
https://api.github.com/users/gmongaras/events{/privacy}
https://api.github.com/users/gmongaras/followers
https://api.github.com/users/gmongaras/following{/other_user}
https://api.github.com/users/gmongaras/gists{/gist_id}
https://github.com/gmongaras
gmongaras
MDQ6VXNlcjQzNTAxNzM4
https://api.github.com/users/gmongaras/orgs
https://api.github.com/users/gmongaras/received_events
https://api.github.com/users/gmongaras/repos
https://api.github.com/users/gmongaras/starred{/owner}{/repo}
https://api.github.com/users/gmongaras/subscriptions
User
https://api.github.com/users/gmongaras
public
MEMBER
done in https://github.com/huggingface/datasets/pull/7532 :)
2025-05-07T14:07:34Z
https://github.com/huggingface/datasets/pull/7499#issuecomment-2858741986
https://api.github.com/repos/huggingface/datasets/issues/7499
IC_kwDODunzps6qZPDi
https://api.github.com/repos/huggingface/datasets/issues/comments/2858741986/reactions
2025-05-07T14:07:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2858741986
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
When adding "cache_dir" to datasets.load_dataset, the cache_dir gets lost in the function calls, changing the cache dir to the default path. This fixes a few of these instances.