Push to hub in a pull request
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5528). All of your documentation changes will be reflected on that endpoint.
2023-02-14T10:30:07Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1429492805
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5VNFRF
https://api.github.com/repos/huggingface/datasets/issues/comments/1429492805/reactions
2023-02-14T10:30:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1429492805
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
It seems that the parameter `create_pr` is available for [`0.8.0`](https://huggingface.co/docs/huggingface_hub/v0.8.1/en/package_reference/hf_api#huggingface_hub.HfApi.upload_file) (its not here: [`0.7.0`](https://huggingface.co/docs/huggingface_hub/v0.7.0.rc0/en/package_reference/hf_api#huggingface_hub.HfApi.upload_file)) and onwards. I included a warning, informing the user that no PR was created.
2023-02-14T15:22:22Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1429922836
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5VOuQU
https://api.github.com/repos/huggingface/datasets/issues/comments/1429922836/reactions
2023-02-14T15:22:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1429922836
https://avatars.githubusercontent.com/u/38854604?v=4
https://api.github.com/users/AJDERS/events{/privacy}
https://api.github.com/users/AJDERS/followers
https://api.github.com/users/AJDERS/following{/other_user}
https://api.github.com/users/AJDERS/gists{/gist_id}
https://github.com/AJDERS
AJDERS
MDQ6VXNlcjM4ODU0NjA0
https://api.github.com/users/AJDERS/orgs
https://api.github.com/users/AJDERS/received_events
https://api.github.com/users/AJDERS/repos
https://api.github.com/users/AJDERS/starred{/owner}{/repo}
https://api.github.com/users/AJDERS/subscriptions
User
https://api.github.com/users/AJDERS
public
NONE
@nateraw you are completely right! Actually, the dataset shards is never added to the created pr, only the metadata, as the code is now. Ill look into you suggestion asap. Thank!
2023-02-15T07:07:06Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1430856911
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5VSSTP
https://api.github.com/repos/huggingface/datasets/issues/comments/1430856911/reactions
2023-02-15T07:07:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1430856911
https://avatars.githubusercontent.com/u/38854604?v=4
https://api.github.com/users/AJDERS/events{/privacy}
https://api.github.com/users/AJDERS/followers
https://api.github.com/users/AJDERS/following{/other_user}
https://api.github.com/users/AJDERS/gists{/gist_id}
https://github.com/AJDERS
AJDERS
MDQ6VXNlcjM4ODU0NjA0
https://api.github.com/users/AJDERS/orgs
https://api.github.com/users/AJDERS/received_events
https://api.github.com/users/AJDERS/repos
https://api.github.com/users/AJDERS/starred{/owner}{/repo}
https://api.github.com/users/AJDERS/subscriptions
User
https://api.github.com/users/AJDERS
public
CONTRIBUTOR
@nateraw Nothing more to add, that's a perfect usage of `huggingface_hub` as far as I can tell ! :fire: 

A very nit improvement would be to use the [for .. else ... python statement](https://book.pythontips.com/en/latest/for_-_else.html).
i.e:

```py
if create_pr is True and revision is not None:
    for discussion in get_repo_discussions(repo_id, repo_type='dataset'):
        if discussion.is_pull_request and discussion.git_reference == revision:
            create_pr = False
            break
    else:
        raise ValueError("Provided revision not found")
```
No need for the `revision_found` temporary flag when do so. Yeah ok, it's niche :wink: 
2023-02-15T08:46:02Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1430952792
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5VSptY
https://api.github.com/repos/huggingface/datasets/issues/comments/1430952792/reactions
2023-02-15T08:46:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1430952792
https://avatars.githubusercontent.com/u/11801849?v=4
https://api.github.com/users/Wauplin/events{/privacy}
https://api.github.com/users/Wauplin/followers
https://api.github.com/users/Wauplin/following{/other_user}
https://api.github.com/users/Wauplin/gists{/gist_id}
https://github.com/Wauplin
Wauplin
MDQ6VXNlcjExODAxODQ5
https://api.github.com/users/Wauplin/orgs
https://api.github.com/users/Wauplin/received_events
https://api.github.com/users/Wauplin/repos
https://api.github.com/users/Wauplin/starred{/owner}{/repo}
https://api.github.com/users/Wauplin/subscriptions
User
https://api.github.com/users/Wauplin
public
NONE
I added the suggestions from @nateraw and @Wauplin .
2023-02-15T09:22:15Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1431001509
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5VS1ml
https://api.github.com/repos/huggingface/datasets/issues/comments/1431001509/reactions
2023-02-15T09:22:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1431001509
https://avatars.githubusercontent.com/u/38854604?v=4
https://api.github.com/users/AJDERS/events{/privacy}
https://api.github.com/users/AJDERS/followers
https://api.github.com/users/AJDERS/following{/other_user}
https://api.github.com/users/AJDERS/gists{/gist_id}
https://github.com/AJDERS
AJDERS
MDQ6VXNlcjM4ODU0NjA0
https://api.github.com/users/AJDERS/orgs
https://api.github.com/users/AJDERS/received_events
https://api.github.com/users/AJDERS/repos
https://api.github.com/users/AJDERS/starred{/owner}{/repo}
https://api.github.com/users/AJDERS/subscriptions
User
https://api.github.com/users/AJDERS
public
NONE
> Thanks. Some comments/suggestions below...
> 
> Why have you removed the test for create_pr? You could add it again and just add a pytest skipif when version of huggingface_hub is lower than 0.8.1.

I have added the test again. I removed it because i kept getting errors when calling `create_pull_request` with `repo_id=ds_name` where `temporary_repo = ds_name`, and thought i might look more thoroughly at it later. I have added a test called `test_test` showing this, it gives:
```
tests/test_upstream_hub.py:360: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:124: in _inner_fn
    return fn(*args, **kwargs)
.venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py:3451: in create_pull_request
    return self.create_discussion(
.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:124: in _inner_fn
    return fn(*args, **kwargs)
.venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py:3393: in create_discussion
    hf_raise_for_status(resp)
(...)
E               huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-63ecd2cb-2cf2557a332c86ad27f687b3)
E               
E               Repository Not Found for url: https://huggingface.co/api/models/__DUMMY_TRANSFORMERS_USER__/test-16764648321590/discussions.
E               Please make sure you specified the correct `repo_id` and `repo_type`.
E               If you are trying to access a private or gated repo, make sure you are authenticated.
E               Invalid username or password.
```
2023-02-15T12:57:23Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1431327143
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5VUFGn
https://api.github.com/repos/huggingface/datasets/issues/comments/1431327143/reactions
2023-02-15T12:57:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1431327143
https://avatars.githubusercontent.com/u/38854604?v=4
https://api.github.com/users/AJDERS/events{/privacy}
https://api.github.com/users/AJDERS/followers
https://api.github.com/users/AJDERS/following{/other_user}
https://api.github.com/users/AJDERS/gists{/gist_id}
https://github.com/AJDERS
AJDERS
MDQ6VXNlcjM4ODU0NjA0
https://api.github.com/users/AJDERS/orgs
https://api.github.com/users/AJDERS/received_events
https://api.github.com/users/AJDERS/repos
https://api.github.com/users/AJDERS/starred{/owner}{/repo}
https://api.github.com/users/AJDERS/subscriptions
User
https://api.github.com/users/AJDERS
public
NONE
> > Thanks. Some comments/suggestions below...
> > Why have you removed the test for create_pr? You could add it again and just add a pytest skipif when version of huggingface_hub is lower than 0.8.1.
> 
> I have added the test again. I removed it because i kept getting errors when calling `create_pull_request` with `repo_id=ds_name` where `temporary_repo = ds_name`, and thought i might look more thoroughly at it later. I have added a test called `test_test` showing this, it gives:
> 
> ```
> tests/test_upstream_hub.py:360: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> .venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:124: in _inner_fn
>     return fn(*args, **kwargs)
> .venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py:3451: in create_pull_request
>     return self.create_discussion(
> .venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:124: in _inner_fn
>     return fn(*args, **kwargs)
> .venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py:3393: in create_discussion
>     hf_raise_for_status(resp)
> (...)
> E               huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-63ecd2cb-2cf2557a332c86ad27f687b3)
> E               
> E               Repository Not Found for url: https://huggingface.co/api/models/__DUMMY_TRANSFORMERS_USER__/test-16764648321590/discussions.
> E               Please make sure you specified the correct `repo_id` and `repo_type`.
> E               If you are trying to access a private or gated repo, make sure you are authenticated.
> E               Invalid username or password.
> ```

@albertvillanova, @lhoestq , FYI I have looked at this again, and i haven't figured it out, so the test`test_push_dataset_to_hub_with_pull_request` and the minimal example `test_test` are still failing locally, while the other tests succeed. Do you have any advice?
2023-02-16T15:05:45Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1433231595
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5VbWDr
https://api.github.com/repos/huggingface/datasets/issues/comments/1433231595/reactions
2023-02-16T15:09:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1433231595
https://avatars.githubusercontent.com/u/38854604?v=4
https://api.github.com/users/AJDERS/events{/privacy}
https://api.github.com/users/AJDERS/followers
https://api.github.com/users/AJDERS/following{/other_user}
https://api.github.com/users/AJDERS/gists{/gist_id}
https://github.com/AJDERS
AJDERS
MDQ6VXNlcjM4ODU0NjA0
https://api.github.com/users/AJDERS/orgs
https://api.github.com/users/AJDERS/received_events
https://api.github.com/users/AJDERS/repos
https://api.github.com/users/AJDERS/starred{/owner}{/repo}
https://api.github.com/users/AJDERS/subscriptions
User
https://api.github.com/users/AJDERS
public
NONE
I tried to move all of the "create pr safely"-logic to a seperate function in `_hf_hub_fixes`. I looked at how the exceptions were raised before `huggingface_hub.utils.RepositoryNotFoundError`existed, and make changes accordingly. 
2023-02-16T17:05:20Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1433417636
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5VcDek
https://api.github.com/repos/huggingface/datasets/issues/comments/1433417636/reactions
2023-02-16T17:05:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1433417636
https://avatars.githubusercontent.com/u/38854604?v=4
https://api.github.com/users/AJDERS/events{/privacy}
https://api.github.com/users/AJDERS/followers
https://api.github.com/users/AJDERS/following{/other_user}
https://api.github.com/users/AJDERS/gists{/gist_id}
https://github.com/AJDERS
AJDERS
MDQ6VXNlcjM4ODU0NjA0
https://api.github.com/users/AJDERS/orgs
https://api.github.com/users/AJDERS/received_events
https://api.github.com/users/AJDERS/repos
https://api.github.com/users/AJDERS/starred{/owner}{/repo}
https://api.github.com/users/AJDERS/subscriptions
User
https://api.github.com/users/AJDERS
public
NONE
`create_pr` was set during `push_to_hub`, even though it was `None` from the outset, hence causing tests to fail for older versions of `huggingface_hub`. This is now fixed.

With the implementation of `_hf_hub_fixes.upload_file` the function call expected `commit_message`, `commit_description`. If these are not set we call the function without them, even though we are on a version of `huggingface_hub` where they are not available in `upload_file`.

When `huggingface_hub < 0.5.0` we assume `repo_id` of them form `organisation/name`, so now that we are calling `create_repo` in the tests with `repo_id` not of this form, we need to handle this case, this is now done.

Many tests failed for `dataset_dict` for the above reasons, so the fixes from `arrow_dataset.py` were also added to `dataset_dict.py`. 

**All tests are now passing locally for `huggingface_hub==0.2.0` and `huggingface_hub==0.12.1`…** Im sorry I should have downgraded and went through this a long time ago, but I didn’t realise the extend of these version fixes until recently…
2023-02-20T13:36:06Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1437036971
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5Vp3Gr
https://api.github.com/repos/huggingface/datasets/issues/comments/1437036971/reactions
2023-02-20T13:36:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1437036971
https://avatars.githubusercontent.com/u/38854604?v=4
https://api.github.com/users/AJDERS/events{/privacy}
https://api.github.com/users/AJDERS/followers
https://api.github.com/users/AJDERS/following{/other_user}
https://api.github.com/users/AJDERS/gists{/gist_id}
https://github.com/AJDERS
AJDERS
MDQ6VXNlcjM4ODU0NjA0
https://api.github.com/users/AJDERS/orgs
https://api.github.com/users/AJDERS/received_events
https://api.github.com/users/AJDERS/repos
https://api.github.com/users/AJDERS/starred{/owner}{/repo}
https://api.github.com/users/AJDERS/subscriptions
User
https://api.github.com/users/AJDERS
public
MEMBER
Hi ! FYI bumped the `huggingface-hub` dependency to 0.11 and removed the `_hf_hub_fixes.py` - which should make this PR much easier
2023-03-21T14:32:11Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1477947203
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5YF69D
https://api.github.com/repos/huggingface/datasets/issues/comments/1477947203/reactions
2023-03-21T14:32:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1477947203
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Just now finding this - seems like a cool issue to contribute to. If any more help is needed please ping me! @AJDERS 
2023-10-06T21:58:02Z
https://github.com/huggingface/datasets/pull/5528#issuecomment-1751444830
https://api.github.com/repos/huggingface/datasets/issues/5528
IC_kwDODunzps5oZO1e
https://api.github.com/repos/huggingface/datasets/issues/comments/1751444830/reactions
2023-10-06T21:58:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1751444830
https://avatars.githubusercontent.com/u/55398496?v=4
https://api.github.com/users/abzdel/events{/privacy}
https://api.github.com/users/abzdel/followers
https://api.github.com/users/abzdel/following{/other_user}
https://api.github.com/users/abzdel/gists{/gist_id}
https://github.com/abzdel
abzdel
MDQ6VXNlcjU1Mzk4NDk2
https://api.github.com/users/abzdel/orgs
https://api.github.com/users/abzdel/received_events
https://api.github.com/users/abzdel/repos
https://api.github.com/users/abzdel/starred{/owner}{/repo}
https://api.github.com/users/abzdel/subscriptions
User
https://api.github.com/users/abzdel
public
Fixes #5492.

Introduce new kwarg `create_pr` in `push_to_hub`, which is passed to `HFapi.upload_file`.