Memory leak when streaming
NONE
I seem to have encountered the same problem when loading non streaming datasets. load_from_disk. Causing hundreds of GB of memory, but the dataset actually only has 50GB
2024-11-07T15:53:33Z
https://github.com/huggingface/datasets/issues/7269#issuecomment-2462591466
https://api.github.com/repos/huggingface/datasets/issues/7269
IC_kwDODunzps6SyCnq
https://api.github.com/repos/huggingface/datasets/issues/comments/2462591466/reactions
2024-11-07T15:53:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2462591466
https://avatars.githubusercontent.com/u/14285786?v=4
https://api.github.com/users/enze5088/events{/privacy}
https://api.github.com/users/enze5088/followers
https://api.github.com/users/enze5088/following{/other_user}
https://api.github.com/users/enze5088/gists{/gist_id}
https://github.com/enze5088
enze5088
MDQ6VXNlcjE0Mjg1Nzg2
https://api.github.com/users/enze5088/orgs
https://api.github.com/users/enze5088/received_events
https://api.github.com/users/enze5088/repos
https://api.github.com/users/enze5088/starred{/owner}{/repo}
https://api.github.com/users/enze5088/subscriptions
User
https://api.github.com/users/enze5088
public
MEMBER
FYI when streaming parquet data, only one row group per worker is loaded in memory at a time.

Btw for datasets of embeddings you can surely optimize your RAM by reading the data as torch tensors directly instead of the default python lists

```python
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("WaveGenAI/dataset", streaming=True).with_format("torch")

dataloader = DataLoader(dataset["train"], num_workers=3)
```
2024-11-18T11:46:06Z
https://github.com/huggingface/datasets/issues/7269#issuecomment-2482810043
https://api.github.com/repos/huggingface/datasets/issues/7269
IC_kwDODunzps6T_Ky7
https://api.github.com/repos/huggingface/datasets/issues/comments/2482810043/reactions
2024-11-18T11:46:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2482810043
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Describe the bug

I try to use a dataset with streaming=True, the issue I have is that the RAM usage becomes higher and higher until it is no longer sustainable.

I understand that huggingface store data in ram during the streaming, and more worker in dataloader there are, more a lot of shard will be stored in ram, but the issue I have is that the ram usage is not constant. So after each new shard loaded, the ram usage will be higher and higher.

### Steps to reproduce the bug

You can run this code and see you ram usage, after each shard of 255 examples, your ram usage will be extended.
```py
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("WaveGenAI/dataset", streaming=True)

dataloader = DataLoader(dataset["train"], num_workers=3)

for i, data in enumerate(dataloader):
    print(i, end="\r")
```

### Expected behavior

The Ram usage should be always the same (just 3 shards loaded in the ram).

### Environment info

- `datasets` version: 3.0.1
- Platform: Linux-6.10.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1