[FR] `map` should reuse unchanged columns from the previous dataset to avoid disk usage
COLLABORATOR
You can use the `remove_columns` parameter in `map` to avoid duplicating the columns (and save disk space) and then concatenate the original dataset with the map result:
```python
from datasets import concatenate_datasets
# dummy example
ds_new = ds.map(lambda x: {"new_col": x["col"] + 2}, remove_columns=ds.column_names)
ds_combined = concatenate_datasets([ds, ds_new], axis=1)
```

Doing this automatically is hard to implement efficiently unless we know ahead of time which existing columns will be modified by a `map` transform. We have this info when `input_columns` are specified, so I think this is the only case we can optimize.
2023-07-10T15:21:56Z
https://github.com/huggingface/datasets/issues/6013#issuecomment-1629183653
https://api.github.com/repos/huggingface/datasets/issues/6013
IC_kwDODunzps5hG16l
https://api.github.com/repos/huggingface/datasets/issues/comments/1629183653/reactions
2023-07-10T15:37:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1629183653
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Hi @mariosasko 👋 I’d like to start working on this issue.
2025-06-19T06:30:37Z
https://github.com/huggingface/datasets/issues/6013#issuecomment-2986796204
https://api.github.com/repos/huggingface/datasets/issues/6013
IC_kwDODunzps6yBuSs
https://api.github.com/repos/huggingface/datasets/issues/comments/2986796204/reactions
2025-06-19T06:30:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2986796204
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
### Feature request

Currently adding a new column with `map` will cause all the data in the dataset to be duplicated and stored/cached on the disk again. It should reuse unchanged columns. 

### Motivation

This allows having datasets with different columns but sharing some basic columns. Currently, these datasets would become too expensive to store and one would need some kind of on-the-fly join; which also doesn't seem implemented.

### Your contribution

_