[Bug]504 Server Error when loading dataset which was already cached
NONE
I am experiencing the same problem with the following environment:

* `datasets` version: 2.11.0
* Platform: `Linux 5.19.0-41-generic x86_64 GNU/Linux`
* Python version: `3.8.5`
* Huggingface_hub version: 0.13.3
* PyArrow version: `11.0.0`
* Pandas version: `1.5.3`

Trying to get some diagnostics, I got the following:  

```python
>>> from huggingface_hub import scan_cache_dir
>>> sd = scan_cache_dir()
>>> sd
HFCacheInfo(size_on_disk=0, repos=frozenset(), warnings=[CorruptedCacheException('Repo path is not a directory: /home/myname/.cache/huggingface/hub/version_diffusers_cache.txt')])

```
However, that might also be because I had tried to manually specify the `cache_dir` and that resulted in trying to download the dataset again ... but into a folder one level higher up than it should have.

Note that my issue is with the `huggan/wikiart` dataset, so it is not a dataset-specific issue.
2023-05-09T11:17:57Z
https://github.com/huggingface/datasets/issues/5831#issuecomment-1539984305
https://api.github.com/repos/huggingface/datasets/issues/5831
IC_kwDODunzps5bykux
https://api.github.com/repos/huggingface/datasets/issues/comments/1539984305/reactions
2023-05-09T11:19:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1539984305
https://avatars.githubusercontent.com/u/4920691?v=4
https://api.github.com/users/lucidBrot/events{/privacy}
https://api.github.com/users/lucidBrot/followers
https://api.github.com/users/lucidBrot/following{/other_user}
https://api.github.com/users/lucidBrot/gists{/gist_id}
https://github.com/lucidBrot
lucidBrot
MDQ6VXNlcjQ5MjA2OTE=
https://api.github.com/users/lucidBrot/orgs
https://api.github.com/users/lucidBrot/received_events
https://api.github.com/users/lucidBrot/repos
https://api.github.com/users/lucidBrot/starred{/owner}{/repo}
https://api.github.com/users/lucidBrot/subscriptions
User
https://api.github.com/users/lucidBrot
public
NONE
same problem with a private dataset repo, seems the huggingface hub server got some connection problem?
2023-05-09T11:24:32Z
https://github.com/huggingface/datasets/issues/5831#issuecomment-1539993721
https://api.github.com/repos/huggingface/datasets/issues/5831
IC_kwDODunzps5bynB5
https://api.github.com/repos/huggingface/datasets/issues/comments/1539993721/reactions
2023-05-09T11:24:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1539993721
https://avatars.githubusercontent.com/u/20009381?v=4
https://api.github.com/users/TobiasLee/events{/privacy}
https://api.github.com/users/TobiasLee/followers
https://api.github.com/users/TobiasLee/following{/other_user}
https://api.github.com/users/TobiasLee/gists{/gist_id}
https://github.com/TobiasLee
TobiasLee
MDQ6VXNlcjIwMDA5Mzgx
https://api.github.com/users/TobiasLee/orgs
https://api.github.com/users/TobiasLee/received_events
https://api.github.com/users/TobiasLee/repos
https://api.github.com/users/TobiasLee/starred{/owner}{/repo}
https://api.github.com/users/TobiasLee/subscriptions
User
https://api.github.com/users/TobiasLee
public
NONE
Yes, dataset server seems down for now
2023-05-09T12:13:25Z
https://github.com/huggingface/datasets/issues/5831#issuecomment-1540021365
https://api.github.com/repos/huggingface/datasets/issues/5831
IC_kwDODunzps5bytx1
https://api.github.com/repos/huggingface/datasets/issues/comments/1540021365/reactions
2023-05-09T12:13:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1540021365
https://avatars.githubusercontent.com/u/25608944?v=4
https://api.github.com/users/qmeeus/events{/privacy}
https://api.github.com/users/qmeeus/followers
https://api.github.com/users/qmeeus/following{/other_user}
https://api.github.com/users/qmeeus/gists{/gist_id}
https://github.com/qmeeus
qmeeus
MDQ6VXNlcjI1NjA4OTQ0
https://api.github.com/users/qmeeus/orgs
https://api.github.com/users/qmeeus/received_events
https://api.github.com/users/qmeeus/repos
https://api.github.com/users/qmeeus/starred{/owner}{/repo}
https://api.github.com/users/qmeeus/subscriptions
User
https://api.github.com/users/qmeeus
public
COLLABORATOR
@SingL3 You can avoid this error by setting the [`HF_DATASETS_OFFLINE`](https://huggingface.co/docs/datasets/v2.12.0/en/loading#offline) env variable to 1. By default,  if an internet connection is available, we check whether the cache of a cached dataset is up-to-date.

@lucidBrot `datasets`' cache is still not aligned with `huggigface_hub`'s. We plan to align it eventually.
2023-05-09T14:31:46Z
https://github.com/huggingface/datasets/issues/5831#issuecomment-1540245059
https://api.github.com/repos/huggingface/datasets/issues/5831
IC_kwDODunzps5bzkZD
https://api.github.com/repos/huggingface/datasets/issues/comments/1540245059/reactions
2023-05-09T14:31:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1540245059
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Today we had a big issue affecting the Hugging Face Hub, thus all the `504 Server Error: Gateway Time-out` errors.

It is fixed now and loading your datasets should work as expected.
2023-05-09T15:08:00Z
https://github.com/huggingface/datasets/issues/5831#issuecomment-1540336064
https://api.github.com/repos/huggingface/datasets/issues/5831
IC_kwDODunzps5bz6nA
https://api.github.com/repos/huggingface/datasets/issues/comments/1540336064/reactions
2023-05-09T15:08:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1540336064
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Hi, @albertvillanova.
If there is a locally cached version of datasets or something cache using huggingface_hub, when a network problem(either client or server) occurs, is it a better way to fallback to use the current cached version rather than raise a exception and exit?
2023-05-10T01:46:41Z
https://github.com/huggingface/datasets/issues/5831#issuecomment-1541171067
https://api.github.com/repos/huggingface/datasets/issues/5831
IC_kwDODunzps5b3Gd7
https://api.github.com/repos/huggingface/datasets/issues/comments/1541171067/reactions
2023-05-10T01:48:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1541171067
https://avatars.githubusercontent.com/u/20473466?v=4
https://api.github.com/users/SingL3/events{/privacy}
https://api.github.com/users/SingL3/followers
https://api.github.com/users/SingL3/following{/other_user}
https://api.github.com/users/SingL3/gists{/gist_id}
https://github.com/SingL3
SingL3
MDQ6VXNlcjIwNDczNDY2
https://api.github.com/users/SingL3/orgs
https://api.github.com/users/SingL3/received_events
https://api.github.com/users/SingL3/repos
https://api.github.com/users/SingL3/starred{/owner}{/repo}
https://api.github.com/users/SingL3/subscriptions
User
https://api.github.com/users/SingL3
public
### Describe the bug

I have already cached the dataset using:
```
dataset = load_dataset("databricks/databricks-dolly-15k",
                        cache_dir="/mnt/data/llm/datasets/databricks-dolly-15k")
```
After that, I tried to load it again using the same machine, I got this error:
```
Traceback (most recent call last):
  File "/mnt/home/llm/pythia/train.py", line 16, in <module>
    dataset = load_dataset("databricks/databricks-dolly-15k",
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1773, in load_dataset
    builder_instance = load_dataset_builder(
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1502, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1219, in dataset_module_factory
    raise e1 from None
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1186, in dataset_module_factory
    raise e
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1160, in dataset_module_factory
    dataset_info = hf_api.dataset_info(
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 1667, in dataset_info
    hf_raise_for_status(r)
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 301, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/databricks/databricks-dolly-15k
```

### Steps to reproduce the bug

1. cache the databrick-dolly-15k dataset using load_dataset, setting a cache_dir
2. use load_dataset again, setting the same cache_dir

### Expected behavior

Dataset loaded succuessfully.

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-4.18.0-372.16.1.el8_6.x86_64-x86_64-with-glibc2.27
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3