Issue Downloading Certain Datasets After Setting Custom `HF_ENDPOINT`
NONE
Through debugging, I found a potential solution is to modify the code in the error handling module of `huggingface_hub`: https://github.com/huggingface/huggingface_hub/commit/56d6c798c44e83d2a3167e74c022737d8fcbe822 
2024-03-11T09:09:06Z
https://github.com/huggingface/datasets/issues/6728#issuecomment-1987929914
https://api.github.com/repos/huggingface/datasets/issues/6728
IC_kwDODunzps52fWc6
https://api.github.com/repos/huggingface/datasets/issues/comments/1987929914/reactions
2024-03-11T09:09:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1987929914
https://avatars.githubusercontent.com/u/10057041?v=4
https://api.github.com/users/padeoe/events{/privacy}
https://api.github.com/users/padeoe/followers
https://api.github.com/users/padeoe/following{/other_user}
https://api.github.com/users/padeoe/gists{/gist_id}
https://github.com/padeoe
padeoe
MDQ6VXNlcjEwMDU3MDQx
https://api.github.com/users/padeoe/orgs
https://api.github.com/users/padeoe/received_events
https://api.github.com/users/padeoe/repos
https://api.github.com/users/padeoe/starred{/owner}{/repo}
https://api.github.com/users/padeoe/subscriptions
User
https://api.github.com/users/padeoe
public
NONE
@Wauplin 
2024-03-14T15:21:49Z
https://github.com/huggingface/datasets/issues/6728#issuecomment-1997709639
https://api.github.com/repos/huggingface/datasets/issues/6728
IC_kwDODunzps53EqFH
https://api.github.com/repos/huggingface/datasets/issues/comments/1997709639/reactions
2024-03-14T15:21:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1997709639
https://avatars.githubusercontent.com/u/10057041?v=4
https://api.github.com/users/padeoe/events{/privacy}
https://api.github.com/users/padeoe/followers
https://api.github.com/users/padeoe/following{/other_user}
https://api.github.com/users/padeoe/gists{/gist_id}
https://github.com/padeoe
padeoe
MDQ6VXNlcjEwMDU3MDQx
https://api.github.com/users/padeoe/orgs
https://api.github.com/users/padeoe/received_events
https://api.github.com/users/padeoe/repos
https://api.github.com/users/padeoe/starred{/owner}{/repo}
https://api.github.com/users/padeoe/subscriptions
User
https://api.github.com/users/padeoe
public
CONTRIBUTOR
Thanks for investigating and reporting the bug @padeoe! I've opened a PR in `huggingface_hub`  with your suggested fix! :) https://github.com/huggingface/huggingface_hub/pull/2119
2024-03-15T14:38:31Z
https://github.com/huggingface/datasets/issues/6728#issuecomment-1999808086
https://api.github.com/repos/huggingface/datasets/issues/6728
IC_kwDODunzps53MqZW
https://api.github.com/repos/huggingface/datasets/issues/comments/1999808086/reactions
2024-03-15T14:38:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1999808086
https://avatars.githubusercontent.com/u/11801849?v=4
https://api.github.com/users/Wauplin/events{/privacy}
https://api.github.com/users/Wauplin/followers
https://api.github.com/users/Wauplin/following{/other_user}
https://api.github.com/users/Wauplin/gists{/gist_id}
https://github.com/Wauplin
Wauplin
MDQ6VXNlcjExODAxODQ5
https://api.github.com/users/Wauplin/orgs
https://api.github.com/users/Wauplin/received_events
https://api.github.com/users/Wauplin/repos
https://api.github.com/users/Wauplin/starred{/owner}{/repo}
https://api.github.com/users/Wauplin/subscriptions
User
https://api.github.com/users/Wauplin
public
### Describe the bug

This bug is triggered under the following conditions:

- datasets repo ids without organization names trigger errors, such as `bookcorpus`, `gsm8k`, `wikipedia`, rather than in the form of `A/B`.
- If `HF_ENDPOINT` is set and the hostname is not in the form of `(hub-ci.)?huggingface.co`.
- This issue occurs with `datasets>2.15.0` or `huggingface-hub>0.19.4`. For example, using the latest versions: `datasets==2.18.0` and `huggingface-hub==0.21.4`, 

### Steps to reproduce the bug

the issue can be reproduced with the following code:
1. install specific datasets and huggingface_hub.
```bash
pip install datasets==2.18.0
pip install huggingface_hub==0.21.4
```
2. execute python code.
```Python
import os
os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
from datasets import load_dataset
bookcorpus = load_dataset('bookcorpus', split='train')
```
console output:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/padeoe/.local/lib/python3.10/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/padeoe/.local/lib/python3.10/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/padeoe/.local/lib/python3.10/site-packages/datasets/load.py", line 1879, in dataset_module_factory
    raise e1 from None
  File "/home/padeoe/.local/lib/python3.10/site-packages/datasets/load.py", line 1830, in dataset_module_factory
    with fs.open(f"datasets/{path}/{filename}", "r", encoding="utf-8") as f:
  File "/home/padeoe/.local/lib/python3.10/site-packages/fsspec/spec.py", line 1295, in open
    self.open(
  File "/home/padeoe/.local/lib/python3.10/site-packages/fsspec/spec.py", line 1307, in open
    f = self._open(
  File "/home/padeoe/.local/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 228, in _open
    return HfFileSystemFile(self, path, mode=mode, revision=revision, block_size=block_size, **kwargs)
  File "/home/padeoe/.local/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 615, in __init__
    self.resolved_path = fs.resolve_path(path, revision=revision)
  File "/home/padeoe/.local/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 180, in resolve_path
    repo_and_revision_exist, err = self._repo_and_revision_exist(repo_type, repo_id, revision)
  File "/home/padeoe/.local/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 117, in _repo_and_revision_exist
    self._api.repo_info(repo_id, revision=revision, repo_type=repo_type)
  File "/home/padeoe/.local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/padeoe/.local/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2413, in repo_info
    return method(
  File "/home/padeoe/.local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/padeoe/.local/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2286, in dataset_info
    hf_raise_for_status(r)
  File "/home/padeoe/.local/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 362, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 401 Client Error: Unauthorized for url: https://hf-mirror.com/api/datasets/bookcorpus/bookcorpus.py (Request ID: Root=1-65ee8659-5ab10eec5960c63e71f2bb58;b00bdbea-fd6e-4a74-8fe0-bc4682ae090e)
```

### Expected behavior

The dataset was downloaded correctly without any errors.

### Environment info

datasets==2.18.0
huggingface-hub==0.21.4