`push_to_hub` is not concurrency safe (dataset schema corruption)
NONE
@lhoestq can you please take a look? I've submitted a PR that fixes this issue. Thanks.
2025-06-09T22:30:31Z
https://github.com/huggingface/datasets/issues/7600#issuecomment-2957183401
https://api.github.com/repos/huggingface/datasets/issues/7600
IC_kwDODunzps6wQwmp
https://api.github.com/repos/huggingface/datasets/issues/comments/2957183401/reactions
2025-06-09T22:30:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2957183401
https://avatars.githubusercontent.com/u/391004?v=4
https://api.github.com/users/sharvil/events{/privacy}
https://api.github.com/users/sharvil/followers
https://api.github.com/users/sharvil/following{/other_user}
https://api.github.com/users/sharvil/gists{/gist_id}
https://github.com/sharvil
sharvil
MDQ6VXNlcjM5MTAwNA==
https://api.github.com/users/sharvil/orgs
https://api.github.com/users/sharvil/received_events
https://api.github.com/users/sharvil/repos
https://api.github.com/users/sharvil/starred{/owner}{/repo}
https://api.github.com/users/sharvil/subscriptions
User
https://api.github.com/users/sharvil
public
MEMBER
Thanks for the ping ! As I said in https://github.com/huggingface/datasets/pull/7605 there is maybe a more general approach using retries :)
2025-06-11T14:14:04Z
https://github.com/huggingface/datasets/issues/7600#issuecomment-2963009643
https://api.github.com/repos/huggingface/datasets/issues/7600
IC_kwDODunzps6wm_Br
https://api.github.com/repos/huggingface/datasets/issues/comments/2963009643/reactions
2025-06-11T14:14:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2963009643
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Dropping this due to inactivity; we've implemented push_to_hub outside of HF datasets that's concurrency safe. Feel free to use the code I provided as a starting point if there's still interest in addressing this issue.
2025-06-23T19:36:37Z
https://github.com/huggingface/datasets/issues/7600#issuecomment-2997723601
https://api.github.com/repos/huggingface/datasets/issues/7600
IC_kwDODunzps6yraHR
https://api.github.com/repos/huggingface/datasets/issues/comments/2997723601/reactions
2025-06-23T19:36:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2997723601
https://avatars.githubusercontent.com/u/391004?v=4
https://api.github.com/users/sharvil/events{/privacy}
https://api.github.com/users/sharvil/followers
https://api.github.com/users/sharvil/following{/other_user}
https://api.github.com/users/sharvil/gists{/gist_id}
https://github.com/sharvil
sharvil
MDQ6VXNlcjM5MTAwNA==
https://api.github.com/users/sharvil/orgs
https://api.github.com/users/sharvil/received_events
https://api.github.com/users/sharvil/repos
https://api.github.com/users/sharvil/starred{/owner}{/repo}
https://api.github.com/users/sharvil/subscriptions
User
https://api.github.com/users/sharvil
public
MEMBER
Exploring another fix here: https://github.com/huggingface/datasets/issues/7600
2025-07-30T09:33:50Z
https://github.com/huggingface/datasets/issues/7600#issuecomment-3135533318
https://api.github.com/repos/huggingface/datasets/issues/7600
IC_kwDODunzps665HEG
https://api.github.com/repos/huggingface/datasets/issues/comments/3135533318/reactions
2025-07-30T09:33:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3135533318
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Describe the bug

Concurrent processes modifying and pushing a dataset can overwrite each others' dataset card, leaving the dataset unusable.

Consider this scenario:
- we have an Arrow dataset
- there are `N` configs of the dataset
- there are `N` independent processes operating on each of the individual configs (e.g. adding a column, `new_col`)
- each process calls `push_to_hub` on their particular config when they're done processing
- all calls to `push_to_hub` succeed
- the `README.md` now has some configs with `new_col` added and some with `new_col` missing

Any attempt to load a config (using `load_dataset`) where `new_col` is missing will fail because of a schema mismatch between `README.md` and the Arrow files. Fixing the dataset requires updating `README.md` by hand with the correct schema for the affected config. In effect, `push_to_hub` is doing a `git push --force` (I found this behavior quite surprising).

We have hit this issue every time we run processing jobs over our datasets and have to fix corrupted schemas by hand.

Reading through the code, it seems that specifying a [`parent_commit`](https://github.com/huggingface/huggingface_hub/blob/v0.32.4/src/huggingface_hub/hf_api.py#L4587) hash around here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5794 would get us to a normal, non-forced git push, and avoid schema corruption. I'm not familiar enough with the code to know how to determine the commit hash from which the in-memory dataset card was loaded.

### Steps to reproduce the bug

See above.

### Expected behavior

Concurrent edits to disjoint configs of a dataset should never corrupt the dataset schema.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.2
- `fsspec` version: 2023.9.0