`load_dataset` fails to load dataset saved by `save_to_disk`
NONE
In my case the error was:
```
ValueError: You are trying to load a dataset that was saved using `save_to_disk`. Please use `load_from_disk` instead.
```
Did you try `load_from_disk`?
2024-07-23T09:09:16Z
https://github.com/huggingface/datasets/issues/7018#issuecomment-2244670695
https://api.github.com/repos/huggingface/datasets/issues/7018
IC_kwDODunzps6FyvTn
https://api.github.com/repos/huggingface/datasets/issues/comments/2244670695/reactions
2024-07-23T09:09:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2244670695
https://avatars.githubusercontent.com/u/21305646?v=4
https://api.github.com/users/happyTonakai/events{/privacy}
https://api.github.com/users/happyTonakai/followers
https://api.github.com/users/happyTonakai/following{/other_user}
https://api.github.com/users/happyTonakai/gists{/gist_id}
https://github.com/happyTonakai
happyTonakai
MDQ6VXNlcjIxMzA1NjQ2
https://api.github.com/users/happyTonakai/orgs
https://api.github.com/users/happyTonakai/received_events
https://api.github.com/users/happyTonakai/repos
https://api.github.com/users/happyTonakai/starred{/owner}{/repo}
https://api.github.com/users/happyTonakai/subscriptions
User
https://api.github.com/users/happyTonakai
public
CONTRIBUTOR
More generally, any reason there is no API consistency between save_to_disk and push_to_hub ? 

Would be nice to be able to save_to_disk and then upload manually to the hub and load_dataset (which works in some situations but not all)...
2024-08-05T09:21:54Z
https://github.com/huggingface/datasets/issues/7018#issuecomment-2268589975
https://api.github.com/repos/huggingface/datasets/issues/7018
IC_kwDODunzps6HN--X
https://api.github.com/repos/huggingface/datasets/issues/comments/2268589975/reactions
2024-08-05T09:21:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2268589975
https://avatars.githubusercontent.com/u/43467008?v=4
https://api.github.com/users/ManuelFay/events{/privacy}
https://api.github.com/users/ManuelFay/followers
https://api.github.com/users/ManuelFay/following{/other_user}
https://api.github.com/users/ManuelFay/gists{/gist_id}
https://github.com/ManuelFay
ManuelFay
MDQ6VXNlcjQzNDY3MDA4
https://api.github.com/users/ManuelFay/orgs
https://api.github.com/users/ManuelFay/received_events
https://api.github.com/users/ManuelFay/repos
https://api.github.com/users/ManuelFay/starred{/owner}{/repo}
https://api.github.com/users/ManuelFay/subscriptions
User
https://api.github.com/users/ManuelFay
public
NONE
I have the exact same problem !
2024-12-03T11:17:44Z
https://github.com/huggingface/datasets/issues/7018#issuecomment-2514262982
https://api.github.com/repos/huggingface/datasets/issues/7018
IC_kwDODunzps6V3JvG
https://api.github.com/repos/huggingface/datasets/issues/comments/2514262982/reactions
2024-12-03T11:17:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2514262982
https://avatars.githubusercontent.com/u/29790817?v=4
https://api.github.com/users/kfarivar/events{/privacy}
https://api.github.com/users/kfarivar/followers
https://api.github.com/users/kfarivar/following{/other_user}
https://api.github.com/users/kfarivar/gists{/gist_id}
https://github.com/kfarivar
kfarivar
MDQ6VXNlcjI5NzkwODE3
https://api.github.com/users/kfarivar/orgs
https://api.github.com/users/kfarivar/received_events
https://api.github.com/users/kfarivar/repos
https://api.github.com/users/kfarivar/starred{/owner}{/repo}
https://api.github.com/users/kfarivar/subscriptions
User
https://api.github.com/users/kfarivar
public
NONE
`load_from_disk` managed to load the dataset, but the bug with `load_dataset` needs to be fixed. 
2024-12-03T11:26:16Z
https://github.com/huggingface/datasets/issues/7018#issuecomment-2514283062
https://api.github.com/repos/huggingface/datasets/issues/7018
IC_kwDODunzps6V3Oo2
https://api.github.com/repos/huggingface/datasets/issues/comments/2514283062/reactions
2024-12-03T11:26:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2514283062
https://avatars.githubusercontent.com/u/29790817?v=4
https://api.github.com/users/kfarivar/events{/privacy}
https://api.github.com/users/kfarivar/followers
https://api.github.com/users/kfarivar/following{/other_user}
https://api.github.com/users/kfarivar/gists{/gist_id}
https://github.com/kfarivar
kfarivar
MDQ6VXNlcjI5NzkwODE3
https://api.github.com/users/kfarivar/orgs
https://api.github.com/users/kfarivar/received_events
https://api.github.com/users/kfarivar/repos
https://api.github.com/users/kfarivar/starred{/owner}{/repo}
https://api.github.com/users/kfarivar/subscriptions
User
https://api.github.com/users/kfarivar
public
NONE
any update ? I need some function from load dataset like num_proc, or streaming to optimize ram, ... but got this error
2025-05-24T05:21:11Z
https://github.com/huggingface/datasets/issues/7018#issuecomment-2906442117
https://api.github.com/repos/huggingface/datasets/issues/7018
IC_kwDODunzps6tPMmF
https://api.github.com/repos/huggingface/datasets/issues/comments/2906442117/reactions
2025-05-24T05:21:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2906442117
https://avatars.githubusercontent.com/u/89495585?v=4
https://api.github.com/users/anhalu/events{/privacy}
https://api.github.com/users/anhalu/followers
https://api.github.com/users/anhalu/following{/other_user}
https://api.github.com/users/anhalu/gists{/gist_id}
https://github.com/anhalu
anhalu
MDQ6VXNlcjg5NDk1NTg1
https://api.github.com/users/anhalu/orgs
https://api.github.com/users/anhalu/received_events
https://api.github.com/users/anhalu/repos
https://api.github.com/users/anhalu/starred{/owner}{/repo}
https://api.github.com/users/anhalu/subscriptions
User
https://api.github.com/users/anhalu
public
### Describe the bug

This code fails to load the dataset it just saved:

```python
from datasets import load_dataset
from transformers import AutoTokenizer

MODEL = "google-bert/bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

dataset = load_dataset("yelp_review_full")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets.save_to_disk("dataset")

tokenized_datasets = load_dataset("dataset/")  # raises
```

It raises `ValueError: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('test'): ('json', {})}`.

I believe this bug is caused by the [logic that tries to infer dataset format](https://github.com/huggingface/datasets/blob/9af8dd3de7626183a9a9ec8973cebc672d690400/src/datasets/load.py#L556). It counts the most common file extension. However, a small dataset can fit in a single `.arrow` file and have two JSON metadata files, causing the format to be inferred as JSON:

```shell
$ ls -l dataset/test
-rw-r--r-- 1 sliedes sliedes 191498784 Jul  1 13:55 data-00000-of-00001.arrow
-rw-r--r-- 1 sliedes sliedes      1730 Jul  1 13:55 dataset_info.json
-rw-r--r-- 1 sliedes sliedes       249 Jul  1 13:55 state.json
```

### Steps to reproduce the bug

Execute the code above.

### Expected behavior

The dataset is loaded successfully.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-6.9.3-arch1-1-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0
