Large memory use when loading large datasets to a ZFS pool
MEMBER
Hi ! datasets are memory mapped from disk, so they don't fill out your RAM. Not sure what's the source of your memory issue.

What kind of system are you using ? and what kind of disk ?
2025-05-07T14:41:14Z
https://github.com/huggingface/datasets/issues/7546#issuecomment-2858864190
https://api.github.com/repos/huggingface/datasets/issues/7546
IC_kwDODunzps6qZs4-
https://api.github.com/repos/huggingface/datasets/issues/comments/2858864190/reactions
2025-05-07T14:41:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2858864190
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Well, the fact of the matter is that my RAM is getting filled out by running the given example, as shown in [this video](https://streamable.com/usb0ql).

My system is a GPU server running Ubuntu. The disk is a SATA SSD attached to the server using a backplane. It is formatted with ZFS, mounted in /cache, and my HF_HOME is set to /cache/hf

I really need this fixed, so I am more than willing to test out various suggestions you might have, or write a PR if we can figure out what is going on.
2025-05-08T14:05:27Z
https://github.com/huggingface/datasets/issues/7546#issuecomment-2863196529
https://api.github.com/repos/huggingface/datasets/issues/7546
IC_kwDODunzps6qqOlx
https://api.github.com/repos/huggingface/datasets/issues/comments/2863196529/reactions
2025-05-08T14:05:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2863196529
https://avatars.githubusercontent.com/u/6875946?v=4
https://api.github.com/users/FredHaa/events{/privacy}
https://api.github.com/users/FredHaa/followers
https://api.github.com/users/FredHaa/following{/other_user}
https://api.github.com/users/FredHaa/gists{/gist_id}
https://github.com/FredHaa
FredHaa
MDQ6VXNlcjY4NzU5NDY=
https://api.github.com/users/FredHaa/orgs
https://api.github.com/users/FredHaa/received_events
https://api.github.com/users/FredHaa/repos
https://api.github.com/users/FredHaa/starred{/owner}{/repo}
https://api.github.com/users/FredHaa/subscriptions
User
https://api.github.com/users/FredHaa
public
MEMBER
I'm not super familiar with ZFS, but it looks like it loads the data in memory when the files are memory mapped, which is an issue.

Maybe it's a caching mechanism ? Since `datasets` accesses every memory mapped file to read a small part (the metadata of the arrow record batches), maybe ZFS brings the whole files in memory for quicker subsequent reads. This is an antipattern when it comes to lazy loading datasets of that size though
2025-05-12T14:59:32Z
https://github.com/huggingface/datasets/issues/7546#issuecomment-2872924643
https://api.github.com/repos/huggingface/datasets/issues/7546
IC_kwDODunzps6rPVnj
https://api.github.com/repos/huggingface/datasets/issues/comments/2872924643/reactions
2025-05-12T14:59:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2872924643
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
This is the answer.

I tried changing my HF_HOME to an NFS share, and no RAM is then consumed loading the dataset.

I will try to see if I can find a way to configure the ZFS pool to not cache the files (disabling the ARC/primary cache didn't work), and if I do write the solution in this issue. If I can't I guess I have to reformat my cache drive.
2025-05-13T13:29:53Z
https://github.com/huggingface/datasets/issues/7546#issuecomment-2876535316
https://api.github.com/repos/huggingface/datasets/issues/7546
IC_kwDODunzps6rdHIU
https://api.github.com/repos/huggingface/datasets/issues/comments/2876535316/reactions
2025-05-13T13:29:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2876535316
https://avatars.githubusercontent.com/u/6875946?v=4
https://api.github.com/users/FredHaa/events{/privacy}
https://api.github.com/users/FredHaa/followers
https://api.github.com/users/FredHaa/following{/other_user}
https://api.github.com/users/FredHaa/gists{/gist_id}
https://github.com/FredHaa
FredHaa
MDQ6VXNlcjY4NzU5NDY=
https://api.github.com/users/FredHaa/orgs
https://api.github.com/users/FredHaa/received_events
https://api.github.com/users/FredHaa/repos
https://api.github.com/users/FredHaa/starred{/owner}{/repo}
https://api.github.com/users/FredHaa/subscriptions
User
https://api.github.com/users/FredHaa
public
### Describe the bug

When I load large parquet based datasets from the hub like `MLCommons/peoples_speech` using `load_dataset`, all my memory (500GB) is used and isn't released after loading, meaning that the process is terminated by the kernel if I try to load an additional dataset. This makes it impossible to train models using multiple large datasets.

### Steps to reproduce the bug

`uv run --with datasets==3.5.1 python`

```python
from datasets import load_dataset
load_dataset('MLCommons/peoples_speech', 'clean')
load_dataset('mozilla-foundation/common_voice_17_0', 'en')
```

### Expected behavior

I would expect that a lot less than 500GB of RAM would be required to load the dataset, or at least that the RAM usage would be cleared as soon as the dataset is loaded (and thus reside as a memory mapped file) such that other datasets can be loaded.

### Environment info

I am currently using the latest datasets==3.5.1 but I have had the same problem with multiple other versions.