HF_DATASETS_CACHE ignored?
NONE
FWIW, it does eventually write to /tmp/roller/datasets when generating the final version.
2025-03-26T17:39:11Z
https://github.com/huggingface/datasets/issues/7480#issuecomment-2755246164
https://api.github.com/repos/huggingface/datasets/issues/7480
IC_kwDODunzps6kObhU
https://api.github.com/repos/huggingface/datasets/issues/comments/2755246164/reactions
2025-03-26T17:39:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2755246164
https://avatars.githubusercontent.com/u/31896?v=4
https://api.github.com/users/stephenroller/events{/privacy}
https://api.github.com/users/stephenroller/followers
https://api.github.com/users/stephenroller/following{/other_user}
https://api.github.com/users/stephenroller/gists{/gist_id}
https://github.com/stephenroller
stephenroller
MDQ6VXNlcjMxODk2
https://api.github.com/users/stephenroller/orgs
https://api.github.com/users/stephenroller/received_events
https://api.github.com/users/stephenroller/repos
https://api.github.com/users/stephenroller/starred{/owner}{/repo}
https://api.github.com/users/stephenroller/subscriptions
User
https://api.github.com/users/stephenroller
public
CONTRIBUTOR
Hey, I’d love to work on this issue but I am a beginner, can I work it with you?
2025-04-07T16:47:37Z
https://github.com/huggingface/datasets/issues/7480#issuecomment-2783983767
https://api.github.com/repos/huggingface/datasets/issues/7480
IC_kwDODunzps6l8DiX
https://api.github.com/repos/huggingface/datasets/issues/comments/2783983767/reactions
2025-04-07T16:47:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2783983767
https://avatars.githubusercontent.com/u/129883215?v=4
https://api.github.com/users/Harry-Yang0518/events{/privacy}
https://api.github.com/users/Harry-Yang0518/followers
https://api.github.com/users/Harry-Yang0518/following{/other_user}
https://api.github.com/users/Harry-Yang0518/gists{/gist_id}
https://github.com/Harry-Yang0518
Harry-Yang0518
U_kgDOB73cTw
https://api.github.com/users/Harry-Yang0518/orgs
https://api.github.com/users/Harry-Yang0518/received_events
https://api.github.com/users/Harry-Yang0518/repos
https://api.github.com/users/Harry-Yang0518/starred{/owner}{/repo}
https://api.github.com/users/Harry-Yang0518/subscriptions
User
https://api.github.com/users/Harry-Yang0518
public
CONTRIBUTOR
Hi @lhoestq,
I'd like to look into this issue but I'm still learning. Could you share any quick pointers on the HF_DATASETS_CACHE behavior here? Thanks!
2025-04-07T17:23:56Z
https://github.com/huggingface/datasets/issues/7480#issuecomment-2784067338
https://api.github.com/repos/huggingface/datasets/issues/7480
IC_kwDODunzps6l8X8K
https://api.github.com/repos/huggingface/datasets/issues/comments/2784067338/reactions
2025-04-07T17:23:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2784067338
https://avatars.githubusercontent.com/u/129883215?v=4
https://api.github.com/users/Harry-Yang0518/events{/privacy}
https://api.github.com/users/Harry-Yang0518/followers
https://api.github.com/users/Harry-Yang0518/following{/other_user}
https://api.github.com/users/Harry-Yang0518/gists{/gist_id}
https://github.com/Harry-Yang0518
Harry-Yang0518
U_kgDOB73cTw
https://api.github.com/users/Harry-Yang0518/orgs
https://api.github.com/users/Harry-Yang0518/received_events
https://api.github.com/users/Harry-Yang0518/repos
https://api.github.com/users/Harry-Yang0518/starred{/owner}{/repo}
https://api.github.com/users/Harry-Yang0518/subscriptions
User
https://api.github.com/users/Harry-Yang0518
public
MEMBER
Hi ! `HF_DATASETS_CACHE` is only for the cache files of the `datasets` library, not for the `huggingface_hub` cache for files downloaded from the Hugging Face Hub.

You should either specify `HF_HOME` (parent cache path for everything HF) or both `HF_DATASETS_CACHE` and `HF_HUB_CACHE`
2025-04-08T13:04:44Z
https://github.com/huggingface/datasets/issues/7480#issuecomment-2786374008
https://api.github.com/repos/huggingface/datasets/issues/7480
IC_kwDODunzps6mFLF4
https://api.github.com/repos/huggingface/datasets/issues/comments/2786374008/reactions
2025-04-08T13:04:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2786374008
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR


Thanks for clarifying, @lhoestq! To make sure I’ve got it right:

1. **HF_DATASETS_CACHE** only controls where the **datasets** library writes its own cache files (e.g. processed shards, Arrow files, etc.).
2. Anything downloaded via **huggingface_hub** (models, tokenizers, raw files) still goes into the Hub cache (by default `~/.cache/huggingface/hub`), unless you set **HF_HUB_CACHE** or the parent **HF_HOME**.

So if you want everything off NFS and onto local disk you have two options:

- **Set both**  
  ```bash
  export HF_DATASETS_CACHE=/tmp/roller/datasets  
  export HF_HUB_CACHE=/tmp/roller/hub
  ```
- **Or set**  
  ```bash
  export HF_HOME=/tmp/roller
  ```
  which will apply to both subdirectories.

Is that correct? And would it make sense to add a note to the docs clarifying the distinction (or even support S3 for the Hub cache in the future)? I’m happy to draft a small docs PR if that would help.
2025-04-27T21:04:56Z
https://github.com/huggingface/datasets/issues/7480#issuecomment-2833640084
https://api.github.com/repos/huggingface/datasets/issues/7480
IC_kwDODunzps6o5eqU
https://api.github.com/repos/huggingface/datasets/issues/comments/2833640084/reactions
2025-04-27T21:04:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2833640084
https://avatars.githubusercontent.com/u/129883215?v=4
https://api.github.com/users/Harry-Yang0518/events{/privacy}
https://api.github.com/users/Harry-Yang0518/followers
https://api.github.com/users/Harry-Yang0518/following{/other_user}
https://api.github.com/users/Harry-Yang0518/gists{/gist_id}
https://github.com/Harry-Yang0518
Harry-Yang0518
U_kgDOB73cTw
https://api.github.com/users/Harry-Yang0518/orgs
https://api.github.com/users/Harry-Yang0518/received_events
https://api.github.com/users/Harry-Yang0518/repos
https://api.github.com/users/Harry-Yang0518/starred{/owner}{/repo}
https://api.github.com/users/Harry-Yang0518/subscriptions
User
https://api.github.com/users/Harry-Yang0518
public
MEMBER
Hi, yes that's correct, thanks for the clarification ! A note in the docs would be welcome, thanks
2025-04-28T10:16:15Z
https://github.com/huggingface/datasets/issues/7480#issuecomment-2834737193
https://api.github.com/repos/huggingface/datasets/issues/7480
IC_kwDODunzps6o9qgp
https://api.github.com/repos/huggingface/datasets/issues/comments/2834737193/reactions
2025-04-28T10:16:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2834737193
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Describe the bug

I'm struggling to get things to respect HF_DATASETS_CACHE.

Rationale: I'm on a system that uses NFS for homedir, so downloading to NFS is expensive, slow, and wastes valuable quota compared to local disk. Instead, it seems to rely mostly on HF_HUB_CACHE.

Current version: 3.2.1dev. In the process of testing 3.4.0

### Steps to reproduce the bug

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]

dump.py:
```python
from datasets import load_dataset
dataset = load_dataset("HuggingFaceFW/fineweb", name="sample-100BT", split="train")
```

Repro steps
```bash
# ensure no cache
$ mv ~/.cache/huggingface ~/.cache/huggingface.bak

$ export HF_DATASETS_CACHE=/tmp/roller/datasets
$ rm -rf ${HF_DATASETS_CACHE}
$ env | grep HF | grep -v TOKEN
HF_DATASETS_CACHE=/tmp/roller/datasets

$ python dump.py
# (omitted for brevity)

# (while downloading) 
$ du -hcs ~/.cache/huggingface/hub
18G     hub
18G     total

# (after downloading)
$ du -hcs ~/.cache/huggingface/hub
```

It's a shame because datasets supports s3 (which I could really use right now) but hub does not.

### Expected behavior

* ~/.cache/huggingface/hub stays empty
* /tmp/roller/datasets becomes full of stuff

### Environment info

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]