Use hfh hf_hub_url function
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5196). All of your documentation changes will be reflected on that endpoint.
2022-11-03T10:12:51Z
https://github.com/huggingface/datasets/pull/5196#issuecomment-1301883250
https://api.github.com/repos/huggingface/datasets/issues/5196
IC_kwDODunzps5NmSly
https://api.github.com/repos/huggingface/datasets/issues/comments/1301883250/reactions
2022-11-03T10:12:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1301883250
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
@lhoestq I think we should first agree if `datasets` can introduce the breaking change of ignoring `config.HUB_DATASETS_URL`: some users may have override this.

If so, I then would suggest to initiate a deprecation cycle.
2022-11-08T14:23:41Z
https://github.com/huggingface/datasets/pull/5196#issuecomment-1307302341
https://api.github.com/repos/huggingface/datasets/issues/5196
IC_kwDODunzps5N69nF
https://api.github.com/repos/huggingface/datasets/issues/comments/1307302341/reactions
2022-11-08T14:23:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1307302341
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
After a discussion with the rest of the datasets team, we agreed we can introduce the breaking change of ignoring `config.HUB_DATASETS_URL`: this will have minimal impact, only for **private Hubs**. We will address eventual possible impacts in the future.

Additionally, we also ignore `config.HUB_DEFAULT_VERSION`.

See explanation in this PR description: https://github.com/huggingface/datasets/pull/5196#issue-1434401646
2022-11-08T15:48:46Z
https://github.com/huggingface/datasets/pull/5196#issuecomment-1307430175
https://api.github.com/repos/huggingface/datasets/issues/5196
IC_kwDODunzps5N7c0f
https://api.github.com/repos/huggingface/datasets/issues/comments/1307430175/reactions
2022-11-08T16:03:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1307430175
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
COLLABORATOR
I'm trying to upgrade datasets to 2.7.0 in https://github.com/huggingface/datasets-server, and the tests fail due to this change. I think it's a breaking change (that was not listed in https://github.com/huggingface/datasets/releases/tag/2.7.0) since code that previously worked (by setting `datasets.config.HUB_DATASETS_URL = CI_HUB_DATASETS_URL` for example) does not work anymore.

I'm not sure what is the correct way to set up the tests; besides setting the env var "HF_ENDPOINT" before launching the tests (which, I think, is not a good way to do: the tests should not depend on the environment).
2022-11-21T14:46:27Z
https://github.com/huggingface/datasets/pull/5196#issuecomment-1322170870
https://api.github.com/repos/huggingface/datasets/issues/5196
IC_kwDODunzps5Ozrn2
https://api.github.com/repos/huggingface/datasets/issues/comments/1322170870/reactions
2022-11-21T14:46:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322170870
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
COLLABORATOR
OK, I re-read this thread, and https://github.com/huggingface/datasets/pull/5196#issuecomment-1307430175 explicitely states that `config.HUB_DATASETS_URL` (as well as `config.HUB_DEFAULT_VERSION`) is now ignored. I was expecting the breaking changes to be listed in the release notes: https://github.com/huggingface/datasets/releases/tag/2.7.0.
2022-11-21T14:54:26Z
https://github.com/huggingface/datasets/pull/5196#issuecomment-1322185988
https://api.github.com/repos/huggingface/datasets/issues/5196
IC_kwDODunzps5OzvUE
https://api.github.com/repos/huggingface/datasets/issues/comments/1322185988/reactions
2022-11-21T14:54:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322185988
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
CONTRIBUTOR
> I'm not sure what is the correct way to set up the tests; besides setting the env var "HF_ENDPOINT" before launching the tests (which, I think, is not a good way to do: the tests should not depend on the environment).

I think the current workaround of settings an env variable before launching the tests is "not so bad" when considering the fact that env variables are evaluated at import time in `huggingface_hub` (and most probable `datasets` as well). I think that when refactoring this in huggingface_hub (https://github.com/huggingface/huggingface_hub/issues/1172) I'll opt for instantiating a `Settings` object (or `Constants`) that contains all the settings variables. This way it will not be possible to import attributes individually + tests would be easier. As I see it, it would be similar to [what `Pydantic` does](https://pydantic-docs.helpmanual.io/usage/settings/) even though we most probably don't want Pydantic as a root dependency just for that. 
2022-11-21T14:54:41Z
https://github.com/huggingface/datasets/pull/5196#issuecomment-1322186423
https://api.github.com/repos/huggingface/datasets/issues/5196
IC_kwDODunzps5Ozva3
https://api.github.com/repos/huggingface/datasets/issues/comments/1322186423/reactions
2022-11-21T14:54:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322186423
https://avatars.githubusercontent.com/u/11801849?v=4
https://api.github.com/users/Wauplin/events{/privacy}
https://api.github.com/users/Wauplin/followers
https://api.github.com/users/Wauplin/following{/other_user}
https://api.github.com/users/Wauplin/gists{/gist_id}
https://github.com/Wauplin
Wauplin
MDQ6VXNlcjExODAxODQ5
https://api.github.com/users/Wauplin/orgs
https://api.github.com/users/Wauplin/received_events
https://api.github.com/users/Wauplin/repos
https://api.github.com/users/Wauplin/starred{/owner}{/repo}
https://api.github.com/users/Wauplin/subscriptions
User
https://api.github.com/users/Wauplin
public
MEMBER
You can use fixtures in your tests:
```python
CI_HUB_ENDPOINT = "https://hub-ci.huggingface.co"
CI_HUB_DATASETS_URL = CI_HUB_ENDPOINT + "/datasets/{repo_id}/resolve/{revision}/{path}"
CI_HFH_HUGGINGFACE_CO_URL_TEMPLATE = CI_HUB_ENDPOINT + "/{repo_id}/resolve/{revision}/{filename}"

@pytest.fixture
def ci_hfh_hf_hub_url(monkeypatch):
    monkeypatch.setattr(
        "huggingface_hub.file_download.HUGGINGFACE_CO_URL_TEMPLATE", CI_HFH_HUGGINGFACE_CO_URL_TEMPLATE
    )

@pytest.fixture
def ci_hub_config(monkeypatch):
    monkeypatch.setattr("datasets.config.HF_ENDPOINT", CI_HUB_ENDPOINT)
    monkeypatch.setattr("datasets.config.HUB_DATASETS_URL", CI_HUB_DATASETS_URL)
```

and use `@pytest.fixture(autouse=True)` if you want to always use the CI endpoints.

And when `huggingface-hub` and `datasets` change the way we can set the endpoint, we'll just need to update the fixtures.
I think ultimately you'll only have to change the `huggingface-hub` endpoint settings

2022-11-21T14:57:54Z
https://github.com/huggingface/datasets/pull/5196#issuecomment-1322191056
https://api.github.com/repos/huggingface/datasets/issues/5196
IC_kwDODunzps5OzwjQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1322191056/reactions
2022-11-21T14:57:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322191056
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
OK.

In fact, in datasets-server we set `config.HUB_DATASETS_URL` (https://github.com/huggingface/datasets-server/blob/35a30dbcd687b26db1f02502ea8305f70c064473/workers/splits/src/splits/config.py#L26) at config time, before starting the workers. It's not an issue with how to launch the tests, but with the app in itself.

I understand that for now, the only way to fix this is to setup `HF_ENDPOINT` in the env when launching the app (currently, we set the endpoint with `COMMON_HF_ENDPOINT`, a custom env var I set to be sure not to have side-effects)
2022-11-21T14:58:10Z
https://github.com/huggingface/datasets/pull/5196#issuecomment-1322191411
https://api.github.com/repos/huggingface/datasets/issues/5196
IC_kwDODunzps5Ozwoz
https://api.github.com/repos/huggingface/datasets/issues/comments/1322191411/reactions
2022-12-06T11:38:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322191411
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
COLLABORATOR
> You can use fixtures in your tests:

Thanks, used in https://github.com/huggingface/datasets-server/pull/644.
2022-11-21T15:53:53Z
https://github.com/huggingface/datasets/pull/5196#issuecomment-1322274163
https://api.github.com/repos/huggingface/datasets/issues/5196
IC_kwDODunzps5O0E1z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322274163/reactions
2022-11-21T15:54:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1322274163
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
Small refactoring to use `hf_hub_url` function from `huggingface_hub`.

This PR also creates the `hub` module that will contain all `huggingface_hub` functionalities relevant to `datasets`.

This is a necessary stage before implementing the use of the `hfh` caching system (which uses its `hf_hub_url` under the hood).

EDIT:
~~Finally, we use our `config.HUB_DATASETS_URL` when using `hfh.hf_hub_url`~~
There is a breaking change:  the `hfh` `hf_hub_url` function uses
- `hfh` `HUGGINGFACE_CO_URL_TEMPLATE` URL template, different from the `datasets` `config.HUB_DATASETS_URL`
- also, `hfh` `DEFAULT_REVISION`, instead of `datasets` `config.HUB_DEFAULT_VERSION`