EmptyDatasetError when loading dataset downloaded with HuggingFace cli
COLLABORATOR
Hi! `datasets` is not compatible with `huggingface_hub`'s cache structure, hence the error.

You can track https://github.com/huggingface/datasets/issues/5080 to get notified when this is implemented.
2024-03-15T18:09:01Z
https://github.com/huggingface/datasets/issues/6733#issuecomment-2000187615
https://api.github.com/repos/huggingface/datasets/issues/6733
IC_kwDODunzps53OHDf
https://api.github.com/repos/huggingface/datasets/issues/comments/2000187615/reactions
2024-03-15T18:09:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2000187615
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

I am using a cluster that does not have access to the internet when given a job. I tried downloading the dataset using the huggingface-cli command and then loading it with load_dataset but I get an error:

```raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None```

The dataset I'm using is "lmsys/chatbot_arena_conversations". The folder structure is

- README.md
- data
  - train-00000-of-00001-cced8514c7ed782a.parquet



### Steps to reproduce the bug

1. Download dataset using HuggingFace CLI: ```huggingface-cli download lmsys/chatbot_arena_conversations --local-dir ./lmsys/chatbot_arena_conversations```
2. In Python
```
from datasets import load_dataset
load_dataset("lmsys/chatbot_arena_conversations")
```

### Expected behavior

Should return a Dataset Dict in the form of
```
DatasetDict({
    train: Dataset({
        features: [...],
        num_rows: 33,000
    })
})
```
### Environment info

Python 3.11.5
Datasets 2.18.0
Transformers 4.38.2
Pytorch 2.2.0
Pyarrow 15.0.1
Rocky Linux release 8.9 (Green Obsidian)
