CI is broken for convert_to_parquet: Invalid rev id: refs/pr/1 404 error causes RevisionNotFoundError
MEMBER
Any recent change in the API backend rejecting parameter `revision="refs/pr/1"` to `HfApi.preupload_lfs_files`?
```
f"{endpoint}/api/{repo_type}s/{repo_id}/preupload/{revision}"

https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/test-dataset-5188a8-17219154347516/preupload/refs%2Fpr%2F1.
Invalid rev id: refs/pr/1
```
@Wauplin  @huggingface/datasets @huggingface/moon-landing @huggingface/moon-landing-back 
2024-07-26T08:54:13Z
https://github.com/huggingface/datasets/issues/7073#issuecomment-2252278347
https://api.github.com/repos/huggingface/datasets/issues/7073
IC_kwDODunzps6GPwpL
https://api.github.com/repos/huggingface/datasets/issues/comments/2252278347/reactions
2024-07-26T08:56:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2252278347
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I have temporarily fixed the CI with:
- #7074

However, the underlying issue must be fixed and #7074 must be reverted.
2024-07-26T09:17:02Z
https://github.com/huggingface/datasets/issues/7073#issuecomment-2252319367
https://api.github.com/repos/huggingface/datasets/issues/7073
IC_kwDODunzps6GP6qH
https://api.github.com/repos/huggingface/datasets/issues/comments/2252319367/reactions
2024-07-26T09:17:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2252319367
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Hmm does it do the preupload call before creating the ref cc @Wauplin ?

(in that case it should do a preupload call on the base branch with `?create_pr=1`)
2024-07-26T09:37:24Z
https://github.com/huggingface/datasets/issues/7073#issuecomment-2252356037
https://api.github.com/repos/huggingface/datasets/issues/7073
IC_kwDODunzps6GQDnF
https://api.github.com/repos/huggingface/datasets/issues/comments/2252356037/reactions
2024-07-26T09:37:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2252356037
https://avatars.githubusercontent.com/u/342922?v=4
https://api.github.com/users/coyotte508/events{/privacy}
https://api.github.com/users/coyotte508/followers
https://api.github.com/users/coyotte508/following{/other_user}
https://api.github.com/users/coyotte508/gists{/gist_id}
https://github.com/coyotte508
coyotte508
MDQ6VXNlcjM0MjkyMg==
https://api.github.com/users/coyotte508/orgs
https://api.github.com/users/coyotte508/received_events
https://api.github.com/users/coyotte508/repos
https://api.github.com/users/coyotte508/starred{/owner}{/repo}
https://api.github.com/users/coyotte508/subscriptions
User
https://api.github.com/users/coyotte508
public
MEMBER
@coyotte508, the CI test was implemented 2 months ago and it was working OK until yesterday. See the CI status of the commits in the main branch of `datasets`: https://github.com/huggingface/datasets/commits/main/
2024-07-26T11:08:14Z
https://github.com/huggingface/datasets/issues/7073#issuecomment-2252529431
https://api.github.com/repos/huggingface/datasets/issues/7073
IC_kwDODunzps6GQt8X
https://api.github.com/repos/huggingface/datasets/issues/comments/2252529431/reactions
2024-07-26T11:08:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2252529431
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Yes i get that

We changed the preupload response to return the commit id in https://github.com/huggingface-internal/moon-landing/pull/10756

This line is probably causing the error: https://github.com/huggingface-internal/moon-landing/pull/10756/files#diff-558f6f9865e30bfa091b94d6a4a900138103ddb4eb0bec96b6deec5bf5626fa0R2322

It's weird the error is returned, it means that maybe a ref with 0 history (not even the first commit) was created

Does this change have any impact in production, or just the CI test? If it's just the CI test it should be fixed on your side, if it impacts production we can look at a solution
2024-07-26T12:42:17Z
https://github.com/huggingface/datasets/issues/7073#issuecomment-2252684271
https://api.github.com/repos/huggingface/datasets/issues/7073
IC_kwDODunzps6GRTvv
https://api.github.com/repos/huggingface/datasets/issues/comments/2252684271/reactions
2024-07-26T12:43:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2252684271
https://avatars.githubusercontent.com/u/342922?v=4
https://api.github.com/users/coyotte508/events{/privacy}
https://api.github.com/users/coyotte508/followers
https://api.github.com/users/coyotte508/following{/other_user}
https://api.github.com/users/coyotte508/gists{/gist_id}
https://github.com/coyotte508
coyotte508
MDQ6VXNlcjM0MjkyMg==
https://api.github.com/users/coyotte508/orgs
https://api.github.com/users/coyotte508/received_events
https://api.github.com/users/coyotte508/repos
https://api.github.com/users/coyotte508/starred{/owner}{/repo}
https://api.github.com/users/coyotte508/subscriptions
User
https://api.github.com/users/coyotte508
public
MEMBER
@coyotte508 it impacts production: `convert_to_parquet` raises the above error when the dataset has more that one configs/subsets:
- First subset calls `push_to_hub` with `create_pr=True`
- Second subset uses the `refs/pr/#` returned by the call above, and calls `push_to_hub` with `revision="refs/pr/#"`
2024-07-26T12:50:11Z
https://github.com/huggingface/datasets/issues/7073#issuecomment-2252697965
https://api.github.com/repos/huggingface/datasets/issues/7073
IC_kwDODunzps6GRXFt
https://api.github.com/repos/huggingface/datasets/issues/comments/2252697965/reactions
2024-07-26T12:50:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2252697965
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I tried removing the `mock_commit` call: https://github.com/huggingface/datasets/pull/7076

And the tests seem to work.

So it's probably because the commit is not actually called, it doesn't actually create the pull request on the remote (and the associated `refs/pr/1`). But the `preupload` call is not mocked.

Anyway it shouldn't impact production, since production isn't mocked
2024-07-26T13:52:32Z
https://github.com/huggingface/datasets/issues/7073#issuecomment-2252819072
https://api.github.com/repos/huggingface/datasets/issues/7073
IC_kwDODunzps6GR0qA
https://api.github.com/repos/huggingface/datasets/issues/comments/2252819072/reactions
2024-07-26T13:54:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2252819072
https://avatars.githubusercontent.com/u/342922?v=4
https://api.github.com/users/coyotte508/events{/privacy}
https://api.github.com/users/coyotte508/followers
https://api.github.com/users/coyotte508/following{/other_user}
https://api.github.com/users/coyotte508/gists{/gist_id}
https://github.com/coyotte508
coyotte508
MDQ6VXNlcjM0MjkyMg==
https://api.github.com/users/coyotte508/orgs
https://api.github.com/users/coyotte508/received_events
https://api.github.com/users/coyotte508/repos
https://api.github.com/users/coyotte508/starred{/owner}{/repo}
https://api.github.com/users/coyotte508/subscriptions
User
https://api.github.com/users/coyotte508
public
MEMBER
@coyotte508 thanks a lot for the investigation and sorry for the noise. 
I promise not trying to fix things when I have a slight fever: my head does not work well.

We need indeed to mock `preupload_lfs_files`: before it was not necessary, but now it is.
2024-07-26T14:11:19Z
https://github.com/huggingface/datasets/issues/7073#issuecomment-2252860407
https://api.github.com/repos/huggingface/datasets/issues/7073
IC_kwDODunzps6GR-v3
https://api.github.com/repos/huggingface/datasets/issues/comments/2252860407/reactions
2024-07-26T14:20:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2252860407
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I fixed the test in:
- #7078

Thanks again, @coyotte508.
2024-07-27T05:48:01Z
https://github.com/huggingface/datasets/issues/7073#issuecomment-2253789093
https://api.github.com/repos/huggingface/datasets/issues/7073
IC_kwDODunzps6GVhel
https://api.github.com/repos/huggingface/datasets/issues/comments/2253789093/reactions
2024-07-27T05:48:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2253789093
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
See: https://github.com/huggingface/datasets/actions/runs/10095313567/job/27915185756
```
FAILED tests/test_hub.py::test_convert_to_parquet - huggingface_hub.utils._errors.RevisionNotFoundError: 404 Client Error. (Request ID: Root=1-66a25839-31ce7b475e70e7db1e4d44c2;b0c8870f-d5ef-4bf2-a6ff-0191f3df0f64)

Revision Not Found for url: https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/test-dataset-5188a8-17219154347516/preupload/refs%2Fpr%2F1.
Invalid rev id: refs/pr/1
```
```
 /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/datasets/hub.py:86: in convert_to_parquet
    dataset.push_to_hub(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/datasets/dataset_dict.py:1722: in push_to_hub
    split_additions, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/datasets/arrow_dataset.py:5511: in _push_parquet_shards_to_hub
    api.preupload_lfs_files(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/huggingface_hub/hf_api.py:4231: in preupload_lfs_files
    _fetch_upload_modes(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py:118: in _inner_fn
    return fn(*args, **kwargs)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/huggingface_hub/_commit_api.py:507: in _fetch_upload_modes
    hf_raise_for_status(resp)
```