fix: loading of datasets from Disk(#7373)
NONE
@nepfaff Could you confirm if this fixes the issue for you? I checked Memray, and everything looked good on my end.

Install: `pip install git+https://github.com/sam-hey/datasets.git@fix/concatenate_datasets`

2025-03-31T11:29:22Z
https://github.com/huggingface/datasets/pull/7489#issuecomment-2765947595
https://api.github.com/repos/huggingface/datasets/issues/7489
IC_kwDODunzps6k3QLL
https://api.github.com/repos/huggingface/datasets/issues/comments/2765947595/reactions
2025-03-31T11:30:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2765947595
https://avatars.githubusercontent.com/u/40773225?v=4
https://api.github.com/users/sam-hey/events{/privacy}
https://api.github.com/users/sam-hey/followers
https://api.github.com/users/sam-hey/following{/other_user}
https://api.github.com/users/sam-hey/gists{/gist_id}
https://github.com/sam-hey
sam-hey
MDQ6VXNlcjQwNzczMjI1
https://api.github.com/users/sam-hey/orgs
https://api.github.com/users/sam-hey/received_events
https://api.github.com/users/sam-hey/repos
https://api.github.com/users/sam-hey/starred{/owner}{/repo}
https://api.github.com/users/sam-hey/subscriptions
User
https://api.github.com/users/sam-hey
public
NONE
Will aim to get to this soon. I don't have a rapid testing pipeline setup but need to wait for some AWS nodes to become free
2025-04-02T02:10:50Z
https://github.com/huggingface/datasets/pull/7489#issuecomment-2771106725
https://api.github.com/repos/huggingface/datasets/issues/7489
IC_kwDODunzps6lK7ul
https://api.github.com/repos/huggingface/datasets/issues/comments/2771106725/reactions
2025-04-02T02:10:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2771106725
https://avatars.githubusercontent.com/u/53228351?v=4
https://api.github.com/users/nepfaff/events{/privacy}
https://api.github.com/users/nepfaff/followers
https://api.github.com/users/nepfaff/following{/other_user}
https://api.github.com/users/nepfaff/gists{/gist_id}
https://github.com/nepfaff
nepfaff
MDQ6VXNlcjUzMjI4MzUx
https://api.github.com/users/nepfaff/orgs
https://api.github.com/users/nepfaff/received_events
https://api.github.com/users/nepfaff/repos
https://api.github.com/users/nepfaff/starred{/owner}{/repo}
https://api.github.com/users/nepfaff/subscriptions
User
https://api.github.com/users/nepfaff
public
NONE
I now set up a small experiment:

```python
# Log initial RAM usage
    process = psutil.Process(os.getpid())
    initial_ram = process.memory_info().rss / (1024 * 1024)  # Convert to MB
    logging.info(f"Initial RAM usage: {initial_ram:.2f} MB")

    chunk_datasets = [
        Dataset.load_from_disk(dataset_path, keep_in_memory=False) for _ in range(N)
    ]
    combined_dataset = concatenate_datasets(chunk_datasets)

    # Log final RAM usage
    final_ram = process.memory_info().rss / (1024 * 1024)  # Convert to MB
    ram_diff = final_ram - initial_ram
    logging.info(f"Final RAM usage: {final_ram:.2f} MB")
    logging.info(f"RAM usage increase: {ram_diff:.2f} MB")
```

The RAM usage is linearly correlated with `N` on datasets master!

For my test dataset:
- N=5 => RAM usage increase: 26302.91 MB
- N=10 => RAM usage increase: 52315.18 MB
- N=20 => RAM usage increase: 104510.65 MB
- N=40 => RAM usage increase: 209166.30 MB

Unfortunately, your patch doesn't seem to change this:
```bash
pip install git+https://github.com/sam-hey/datasets.git@fix/concatenate_datasets
pip list | grep datasets
datasets                 3.5.1.dev0
```
Gives exactly the same RAM statistics.

**Edit:** The results are a bit flawed as the memory increase all seems to come from `Dataset.load_from_disk(dataset_path, keep_in_memory=False)` here (which I don't think should happen either?) and not from `concatenate_datasets`. This seems different from my large-scale setup that runs out of memory during `concatenate_datasets` but I don't seem to be able to replicate this here...
2025-04-03T03:04:03Z
https://github.com/huggingface/datasets/pull/7489#issuecomment-2774316091
https://api.github.com/repos/huggingface/datasets/issues/7489
IC_kwDODunzps6lXLQ7
https://api.github.com/repos/huggingface/datasets/issues/comments/2774316091/reactions
2025-04-03T03:41:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2774316091
https://avatars.githubusercontent.com/u/53228351?v=4
https://api.github.com/users/nepfaff/events{/privacy}
https://api.github.com/users/nepfaff/followers
https://api.github.com/users/nepfaff/following{/other_user}
https://api.github.com/users/nepfaff/gists{/gist_id}
https://github.com/nepfaff
nepfaff
MDQ6VXNlcjUzMjI4MzUx
https://api.github.com/users/nepfaff/orgs
https://api.github.com/users/nepfaff/received_events
https://api.github.com/users/nepfaff/repos
https://api.github.com/users/nepfaff/starred{/owner}{/repo}
https://api.github.com/users/nepfaff/subscriptions
User
https://api.github.com/users/nepfaff
public
NONE
Thanks a lot, @nepfaff, for taking a look at this! It seems that `concatenate_datasets()` is fixed with this PR. I can also confirm that loading a large number of files requires significant memory. However, as I understand it, this is expected/a bug since the memory consumption stems from `pa.memory_map()`, which returns a memory-mapped file.

This behavior might be related to this bug: https://github.com/apache/arrow/issues/34423 

<img width="1728" alt="Screenshot 2025-04-03 at 16 01 11" src="https://github.com/user-attachments/assets/475691d8-3aba-4d7e-b8ef-5e7552c70b14" />

2025-04-03T13:59:37Z
https://github.com/huggingface/datasets/pull/7489#issuecomment-2775891915
https://api.github.com/repos/huggingface/datasets/issues/7489
IC_kwDODunzps6ldL_L
https://api.github.com/repos/huggingface/datasets/issues/comments/2775891915/reactions
2025-04-03T14:02:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2775891915
https://avatars.githubusercontent.com/u/40773225?v=4
https://api.github.com/users/sam-hey/events{/privacy}
https://api.github.com/users/sam-hey/followers
https://api.github.com/users/sam-hey/following{/other_user}
https://api.github.com/users/sam-hey/gists{/gist_id}
https://github.com/sam-hey
sam-hey
MDQ6VXNlcjQwNzczMjI1
https://api.github.com/users/sam-hey/orgs
https://api.github.com/users/sam-hey/received_events
https://api.github.com/users/sam-hey/repos
https://api.github.com/users/sam-hey/starred{/owner}{/repo}
https://api.github.com/users/sam-hey/subscriptions
User
https://api.github.com/users/sam-hey
public
MEMBER
Great ! have you tested that it also fixes the memory issue in your case @iamollas ?

Happy to know that it works for you @sam-hey ! Looking forward to merging this
2025-04-24T16:29:17Z
https://github.com/huggingface/datasets/pull/7489#issuecomment-2828212857
https://api.github.com/repos/huggingface/datasets/issues/7489
IC_kwDODunzps6okxp5
https://api.github.com/repos/huggingface/datasets/issues/comments/2828212857/reactions
2025-04-24T16:32:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2828212857
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7489). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2025-04-24T16:36:35Z
https://github.com/huggingface/datasets/pull/7489#issuecomment-2828232020
https://api.github.com/repos/huggingface/datasets/issues/7489
IC_kwDODunzps6ok2VU
https://api.github.com/repos/huggingface/datasets/issues/comments/2828232020/reactions
2025-04-24T16:36:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2828232020
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
Fixes dataset loading from disk by ensuring that memory maps and streams are properly closed. 


For more details, see https://github.com/huggingface/datasets/issues/7373.