Cache only changed columns?
CONTRIBUTOR
so I guess a workaround to this is to simply remove all columns except the ones to cache and then add them back with `concatenate_datasets(..., axis=1)`.
2024-09-05T13:56:12Z
https://github.com/huggingface/datasets/issues/7138#issuecomment-2331753756
https://api.github.com/repos/huggingface/datasets/issues/7138
IC_kwDODunzps6K-70c
https://api.github.com/repos/huggingface/datasets/issues/comments/2331753756/reactions
2024-09-05T13:56:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2331753756
https://avatars.githubusercontent.com/u/37351874?v=4
https://api.github.com/users/Modexus/events{/privacy}
https://api.github.com/users/Modexus/followers
https://api.github.com/users/Modexus/following{/other_user}
https://api.github.com/users/Modexus/gists{/gist_id}
https://github.com/Modexus
Modexus
MDQ6VXNlcjM3MzUxODc0
https://api.github.com/users/Modexus/orgs
https://api.github.com/users/Modexus/received_events
https://api.github.com/users/Modexus/repos
https://api.github.com/users/Modexus/starred{/owner}{/repo}
https://api.github.com/users/Modexus/subscriptions
User
https://api.github.com/users/Modexus
public
MEMBER
yes this is the right workaround. We're keeping the cache like this to make it easier for people to delete intermediate cache files
2024-09-20T13:27:19Z
https://github.com/huggingface/datasets/issues/7138#issuecomment-2363738557
https://api.github.com/repos/huggingface/datasets/issues/7138
IC_kwDODunzps6M48m9
https://api.github.com/repos/huggingface/datasets/issues/comments/2363738557/reactions
2024-09-20T13:27:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2363738557
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Feature request

Cache only the actual changes to the dataset i.e. changed columns.

### Motivation

I realized that caching actually saves the complete dataset again.
This is especially problematic for image datasets if one wants to only change another column e.g. some metadata and then has to save 5 TB again.

### Your contribution

Is this even viable in the current architecture of the package?
I quickly looked into it and it seems it would require significant changes.

I would spend some time looking into this but maybe somebody could help with the feasibility and some plan to implement before spending too much time on it?