Excessive RAM Usage After Dataset Concatenation concatenate_datasets
NONE
![Image](https://github.com/user-attachments/assets/b6f8bcbd-44af-413e-bc06-65380eb0f746)

![Image](https://github.com/user-attachments/assets/a241fcd8-4b62-495c-926c-685f82015dfb)

Adding a img from memray
https://gist.github.com/sam-hey/00c958f13fb0f7b54d17197fe353002f
2025-01-17T07:54:21Z
https://github.com/huggingface/datasets/issues/7373#issuecomment-2597635219
https://api.github.com/repos/huggingface/datasets/issues/7373
IC_kwDODunzps6a1MST
https://api.github.com/repos/huggingface/datasets/issues/comments/2597635219/reactions
2025-01-17T08:05:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2597635219
https://avatars.githubusercontent.com/u/40773225?v=4
https://api.github.com/users/sam-hey/events{/privacy}
https://api.github.com/users/sam-hey/followers
https://api.github.com/users/sam-hey/following{/other_user}
https://api.github.com/users/sam-hey/gists{/gist_id}
https://github.com/sam-hey
sam-hey
MDQ6VXNlcjQwNzczMjI1
https://api.github.com/users/sam-hey/orgs
https://api.github.com/users/sam-hey/received_events
https://api.github.com/users/sam-hey/repos
https://api.github.com/users/sam-hey/starred{/owner}{/repo}
https://api.github.com/users/sam-hey/subscriptions
User
https://api.github.com/users/sam-hey
public
NONE
I'm having the same issue where concatenation seems to use a huge amount of RAM.

```python
# Load all chunks and concatenate them into a final dataset.
        chunk_datasets = [
            Dataset.load_from_disk(file, keep_in_memory=False)
            for file in tqdm(chunk_files, desc="Loading chunk datasets")
        ]
        logging.info("Concatenating chunk datasets...")
        final_dataset = concatenate_datasets(chunk_datasets)
```

This is a real issue for me as the final dataset is a few terabytes in size. I'm using datasets version `3.1.0`. Also tested with version `3.4.1`
2025-03-26T14:41:24Z
https://github.com/huggingface/datasets/issues/7373#issuecomment-2754676665
https://api.github.com/repos/huggingface/datasets/issues/7373
IC_kwDODunzps6kMQe5
https://api.github.com/repos/huggingface/datasets/issues/comments/2754676665/reactions
2025-03-26T21:03:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2754676665
https://avatars.githubusercontent.com/u/53228351?v=4
https://api.github.com/users/nepfaff/events{/privacy}
https://api.github.com/users/nepfaff/followers
https://api.github.com/users/nepfaff/following{/other_user}
https://api.github.com/users/nepfaff/gists{/gist_id}
https://github.com/nepfaff
nepfaff
MDQ6VXNlcjUzMjI4MzUx
https://api.github.com/users/nepfaff/orgs
https://api.github.com/users/nepfaff/received_events
https://api.github.com/users/nepfaff/repos
https://api.github.com/users/nepfaff/starred{/owner}{/repo}
https://api.github.com/users/nepfaff/subscriptions
User
https://api.github.com/users/nepfaff
public
NONE
I did have a short look, the error seems to be from `memory_map` and the stream not being closed. 

https://github.com/huggingface/datasets/blob/5f8d2ad9a1b0bccfd962d998987228addfd5be9f/src/datasets/table.py#L48-L50


Did not have the time to test jet: https://github.com/sam-hey/datasets/tree/fix/concatenate_datasets

I will probably have a better look in a couple of days. 


2025-03-27T17:38:07Z
https://github.com/huggingface/datasets/issues/7373#issuecomment-2758921505
https://api.github.com/repos/huggingface/datasets/issues/7373
IC_kwDODunzps6kcc0h
https://api.github.com/repos/huggingface/datasets/issues/comments/2758921505/reactions
2025-03-27T17:40:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2758921505
https://avatars.githubusercontent.com/u/40773225?v=4
https://api.github.com/users/sam-hey/events{/privacy}
https://api.github.com/users/sam-hey/followers
https://api.github.com/users/sam-hey/following{/other_user}
https://api.github.com/users/sam-hey/gists{/gist_id}
https://github.com/sam-hey
sam-hey
MDQ6VXNlcjQwNzczMjI1
https://api.github.com/users/sam-hey/orgs
https://api.github.com/users/sam-hey/received_events
https://api.github.com/users/sam-hey/repos
https://api.github.com/users/sam-hey/starred{/owner}{/repo}
https://api.github.com/users/sam-hey/subscriptions
User
https://api.github.com/users/sam-hey
public
### Describe the bug

When loading a dataset from disk, concatenating it, and starting the training process, the RAM usage progressively increases until the kernel terminates the process due to excessive memory consumption.

https://github.com/huggingface/datasets/issues/2276

### Steps to reproduce the bug

```python
from datasets import  DatasetDict, concatenate_datasets

dataset = DatasetDict.load_from_disk("data")

...
...

combined_dataset = concatenate_datasets(
        [dataset[split] for split in dataset]
    )

#start SentenceTransformer training
```

### Expected behavior

I would not expect RAM utilization to increase after concatenation. Removing the concatenation step resolves the issue

### Environment info

sentence-transformers==3.1.1
datasets==3.2.0

python3.10