Make `push_to_hub` atomic (#7600)
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7605). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2025-06-11T14:09:40Z
https://github.com/huggingface/datasets/pull/7605#issuecomment-2962994027
https://api.github.com/repos/huggingface/datasets/issues/7605
IC_kwDODunzps6wm7Nr
https://api.github.com/repos/huggingface/datasets/issues/comments/2962994027/reactions
2025-06-11T14:09:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2962994027
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Hi ! unfortunately we can't allow atomic commits for commits with hundreds of files additions (HF would time out)

Maybe an alternative would be to retry if there was a commit in between ? this could be the default behavior as well
2025-06-11T14:13:15Z
https://github.com/huggingface/datasets/pull/7605#issuecomment-2963006746
https://api.github.com/repos/huggingface/datasets/issues/7605
IC_kwDODunzps6wm-Ua
https://api.github.com/repos/huggingface/datasets/issues/comments/2963006746/reactions
2025-06-11T14:13:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2963006746
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for taking a look – much appreciated!

I've verified that commits with up to 20,000 files don't time out and the commit time scales linearly with the number of operations enqueued. It took just under 2 minutes to complete (successfully) the 20k file commit.

The fundamental issue I'm trying to tackle here is dataset corruption: getting into a state where a dataset on the hub cannot be used when downloaded. Non-atomic commits won't get us there, I think. If, for example, 3 of 5 commits complete and the machine/process calling `push_to_hub` has a network, hardware, or other failure that prevents it from completing the rest of the commits (even with retries) we'll now have some pointer files pointing to the new data and others pointing to the old data => corrupted. While this may seem like an unlikely scenario, it's a regular occurrence at scale.

If you still feel strongly that atomic commits are not the right way to go, I can either set it to not be the default or remove it entirely from this PR.

As for retries, it's a good idea. In a non-atomic world, the logic gets more complicated:
- keep an explicit queue of pending add/delete operations
- chunkwise pop from queue and commit with `parent_commit` set to previous chunked commit hash
- if `create_commit` fails:
  - re-fetch README and set `parent_commit` to latest hash for `revision`
  - re-generate dataset card content
  - swap old `CommitOperationAdd` with new one for README in the pending queue
- resume chunkwise committing from the queue as above

Entirely doable, but more involved than I signed up for with this PR.
2025-06-11T15:11:05Z
https://github.com/huggingface/datasets/pull/7605#issuecomment-2963215614
https://api.github.com/repos/huggingface/datasets/issues/7605
IC_kwDODunzps6wnxT-
https://api.github.com/repos/huggingface/datasets/issues/comments/2963215614/reactions
2025-06-11T15:11:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2963215614
https://avatars.githubusercontent.com/u/391004?v=4
https://api.github.com/users/sharvil/events{/privacy}
https://api.github.com/users/sharvil/followers
https://api.github.com/users/sharvil/following{/other_user}
https://api.github.com/users/sharvil/gists{/gist_id}
https://github.com/sharvil
sharvil
MDQ6VXNlcjM5MTAwNA==
https://api.github.com/users/sharvil/orgs
https://api.github.com/users/sharvil/received_events
https://api.github.com/users/sharvil/repos
https://api.github.com/users/sharvil/starred{/owner}{/repo}
https://api.github.com/users/sharvil/subscriptions
User
https://api.github.com/users/sharvil
public
NONE
Just to clarify – setting the `parent_commit` can be separated from making the commit atomic (which is what I'm suggesting by either atomic commits not the default or removing it from this PR). It's crucial to set the parent commit to avoid the read-modify-write race condition on the README schema.
2025-06-11T15:56:58Z
https://github.com/huggingface/datasets/pull/7605#issuecomment-2963367985
https://api.github.com/repos/huggingface/datasets/issues/7605
IC_kwDODunzps6woWgx
https://api.github.com/repos/huggingface/datasets/issues/comments/2963367985/reactions
2025-06-11T15:56:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2963367985
https://avatars.githubusercontent.com/u/391004?v=4
https://api.github.com/users/sharvil/events{/privacy}
https://api.github.com/users/sharvil/followers
https://api.github.com/users/sharvil/following{/other_user}
https://api.github.com/users/sharvil/gists{/gist_id}
https://github.com/sharvil
sharvil
MDQ6VXNlcjM5MTAwNA==
https://api.github.com/users/sharvil/orgs
https://api.github.com/users/sharvil/received_events
https://api.github.com/users/sharvil/repos
https://api.github.com/users/sharvil/starred{/owner}{/repo}
https://api.github.com/users/sharvil/subscriptions
User
https://api.github.com/users/sharvil
public