Incremental dataset (e.g. `.push_to_hub(..., append=True)`)
CONTRIBUTOR
Yea I think waiting for #6269 would be best, or branching from it. For reference, this [PR](https://github.com/LAION-AI/Discord-Scrapers/pull/2) is progressing pretty well which will do similar using the hf hub for our LAION dataset bot https://github.com/LAION-AI/Discord-Scrapers/pull/2. 
2023-10-13T16:05:25Z
https://github.com/huggingface/datasets/issues/6290#issuecomment-1761757677
https://api.github.com/repos/huggingface/datasets/issues/6290
IC_kwDODunzps5pAknt
https://api.github.com/repos/huggingface/datasets/issues/comments/1761757677/reactions
2023-10-13T16:05:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1761757677
https://avatars.githubusercontent.com/u/1981179?v=4
https://api.github.com/users/ZachNagengast/events{/privacy}
https://api.github.com/users/ZachNagengast/followers
https://api.github.com/users/ZachNagengast/following{/other_user}
https://api.github.com/users/ZachNagengast/gists{/gist_id}
https://github.com/ZachNagengast
ZachNagengast
MDQ6VXNlcjE5ODExNzk=
https://api.github.com/users/ZachNagengast/orgs
https://api.github.com/users/ZachNagengast/received_events
https://api.github.com/users/ZachNagengast/repos
https://api.github.com/users/ZachNagengast/starred{/owner}{/repo}
https://api.github.com/users/ZachNagengast/subscriptions
User
https://api.github.com/users/ZachNagengast
public
NONE
Is there any update on this?
2024-07-18T23:37:01Z
https://github.com/huggingface/datasets/issues/6290#issuecomment-2237777864
https://api.github.com/repos/huggingface/datasets/issues/6290
IC_kwDODunzps6FYcfI
https://api.github.com/repos/huggingface/datasets/issues/comments/2237777864/reactions
2024-07-18T23:37:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2237777864
https://avatars.githubusercontent.com/u/8762181?v=4
https://api.github.com/users/nqyy/events{/privacy}
https://api.github.com/users/nqyy/followers
https://api.github.com/users/nqyy/following{/other_user}
https://api.github.com/users/nqyy/gists{/gist_id}
https://github.com/nqyy
nqyy
MDQ6VXNlcjg3NjIxODE=
https://api.github.com/users/nqyy/orgs
https://api.github.com/users/nqyy/received_events
https://api.github.com/users/nqyy/repos
https://api.github.com/users/nqyy/starred{/owner}{/repo}
https://api.github.com/users/nqyy/subscriptions
User
https://api.github.com/users/nqyy
public
NONE
Is there any update on this?
2024-08-31T17:16:45Z
https://github.com/huggingface/datasets/issues/6290#issuecomment-2322975895
https://api.github.com/repos/huggingface/datasets/issues/6290
IC_kwDODunzps6KdcyX
https://api.github.com/repos/huggingface/datasets/issues/comments/2322975895/reactions
2024-08-31T17:16:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2322975895
https://avatars.githubusercontent.com/u/12135272?v=4
https://api.github.com/users/Elfsong/events{/privacy}
https://api.github.com/users/Elfsong/followers
https://api.github.com/users/Elfsong/following{/other_user}
https://api.github.com/users/Elfsong/gists{/gist_id}
https://github.com/Elfsong
Elfsong
MDQ6VXNlcjEyMTM1Mjcy
https://api.github.com/users/Elfsong/orgs
https://api.github.com/users/Elfsong/received_events
https://api.github.com/users/Elfsong/repos
https://api.github.com/users/Elfsong/starred{/owner}{/repo}
https://api.github.com/users/Elfsong/subscriptions
User
https://api.github.com/users/Elfsong
public
CONTRIBUTOR
No update so far on this feature request but for broader context, this announce will help with incremental datasets https://huggingface.co/blog/xethub-joins-hf :)
2024-09-02T08:11:16Z
https://github.com/huggingface/datasets/issues/6290#issuecomment-2324093331
https://api.github.com/repos/huggingface/datasets/issues/6290
IC_kwDODunzps6KhtmT
https://api.github.com/repos/huggingface/datasets/issues/comments/2324093331/reactions
2024-09-02T08:11:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2324093331
https://avatars.githubusercontent.com/u/11801849?v=4
https://api.github.com/users/Wauplin/events{/privacy}
https://api.github.com/users/Wauplin/followers
https://api.github.com/users/Wauplin/following{/other_user}
https://api.github.com/users/Wauplin/gists{/gist_id}
https://github.com/Wauplin
Wauplin
MDQ6VXNlcjExODAxODQ5
https://api.github.com/users/Wauplin/orgs
https://api.github.com/users/Wauplin/received_events
https://api.github.com/users/Wauplin/repos
https://api.github.com/users/Wauplin/starred{/owner}{/repo}
https://api.github.com/users/Wauplin/subscriptions
User
https://api.github.com/users/Wauplin
public
NONE
Still no update? Whats the current recommended way to upload large datasets to the hub? I can't load it all to memory after some limit right
2025-03-12T09:46:26Z
https://github.com/huggingface/datasets/issues/6290#issuecomment-2717277814
https://api.github.com/repos/huggingface/datasets/issues/6290
IC_kwDODunzps6h9l52
https://api.github.com/repos/huggingface/datasets/issues/comments/2717277814/reactions
2025-03-12T09:46:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2717277814
https://avatars.githubusercontent.com/u/19579420?v=4
https://api.github.com/users/tolgadur/events{/privacy}
https://api.github.com/users/tolgadur/followers
https://api.github.com/users/tolgadur/following{/other_user}
https://api.github.com/users/tolgadur/gists{/gist_id}
https://github.com/tolgadur
tolgadur
MDQ6VXNlcjE5NTc5NDIw
https://api.github.com/users/tolgadur/orgs
https://api.github.com/users/tolgadur/received_events
https://api.github.com/users/tolgadur/repos
https://api.github.com/users/tolgadur/starred{/owner}{/repo}
https://api.github.com/users/tolgadur/subscriptions
User
https://api.github.com/users/tolgadur
public
NONE
> Still no update? Whats the current recommended way to upload large datasets to the hub? I can't load it all to memory after some limit right

@tolgadur bro, you may set 'data_dir' as a temporary solution like that:

`Dataset.push_to_hub('your_repo_aame', data_dir='data_{index}')`

2025-03-12T09:52:54Z
https://github.com/huggingface/datasets/issues/6290#issuecomment-2717295764
https://api.github.com/repos/huggingface/datasets/issues/6290
IC_kwDODunzps6h9qSU
https://api.github.com/repos/huggingface/datasets/issues/comments/2717295764/reactions
2025-03-12T09:52:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2717295764
https://avatars.githubusercontent.com/u/12135272?v=4
https://api.github.com/users/Elfsong/events{/privacy}
https://api.github.com/users/Elfsong/followers
https://api.github.com/users/Elfsong/following{/other_user}
https://api.github.com/users/Elfsong/gists{/gist_id}
https://github.com/Elfsong
Elfsong
MDQ6VXNlcjEyMTM1Mjcy
https://api.github.com/users/Elfsong/orgs
https://api.github.com/users/Elfsong/received_events
https://api.github.com/users/Elfsong/repos
https://api.github.com/users/Elfsong/starred{/owner}{/repo}
https://api.github.com/users/Elfsong/subscriptions
User
https://api.github.com/users/Elfsong
public
### Feature request

Have the possibility to do `ds.push_to_hub(..., append=True)`.

### Motivation

Requested in this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65252597c4edc168202a5eaa) and 
this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/4#6524f675c9607bdffb208d8f). Discussed internally on [slack](https://huggingface.slack.com/archives/C02EMARJ65P/p1696950642610639?thread_ts=1690554266.830949&cid=C02EMARJ65P).


### Your contribution

What I suggest to do for parquet datasets is to use `CommitOperationCopy` + `CommitOperationDelete` from `huggingface_hub`:
1. list files
2. copy files from parquet-0001-of-0004 to parquet-0001-of-0005
3. delete files like parquet-0001-of-0004
4. generate + add last parquet file parquet-0005-of-0005

=> make a single commit with all commit operations at once


I think it should be quite straightforward to implement. Happy to review a PR (maybe conflicting with the ongoing "1 commit push_to_hub" PR https://github.com/huggingface/datasets/pull/6269)