Fix data file module inference
MEMBER
Hi ! datasets saved using `save_to_disk` should be loaded with `load_from_disk` ;)
2024-09-02T14:07:05Z
https://github.com/huggingface/datasets/pull/7132#issuecomment-2324840502
https://api.github.com/repos/huggingface/datasets/issues/7132
IC_kwDODunzps6KkkA2
https://api.github.com/repos/huggingface/datasets/issues/comments/2324840502/reactions
2024-09-02T14:07:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2324840502
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
It is convienient to just pass in a path to a local dataset or one from the hub and use the same function to load it. Is it not possible to get this fix merged in to allow this? 
2024-09-02T16:54:25Z
https://github.com/huggingface/datasets/pull/7132#issuecomment-2325101062
https://api.github.com/repos/huggingface/datasets/issues/7132
IC_kwDODunzps6KljoG
https://api.github.com/repos/huggingface/datasets/issues/comments/2325101062/reactions
2024-09-02T16:54:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2325101062
https://avatars.githubusercontent.com/u/1714412?v=4
https://api.github.com/users/HennerM/events{/privacy}
https://api.github.com/users/HennerM/followers
https://api.github.com/users/HennerM/following{/other_user}
https://api.github.com/users/HennerM/gists{/gist_id}
https://github.com/HennerM
HennerM
MDQ6VXNlcjE3MTQ0MTI=
https://api.github.com/users/HennerM/orgs
https://api.github.com/users/HennerM/received_events
https://api.github.com/users/HennerM/repos
https://api.github.com/users/HennerM/starred{/owner}{/repo}
https://api.github.com/users/HennerM/subscriptions
User
https://api.github.com/users/HennerM
public
MEMBER
We can modify `save_to_disk` to write the dataset in a structure supported by the Hub in this case, it's kind of a legacy function anyway
2024-09-02T19:52:12Z
https://github.com/huggingface/datasets/pull/7132#issuecomment-2325249141
https://api.github.com/repos/huggingface/datasets/issues/7132
IC_kwDODunzps6KmHx1
https://api.github.com/repos/huggingface/datasets/issues/comments/2325249141/reactions
2024-09-02T19:52:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2325249141
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I saved a dataset with two splits to disk with `DatasetDict.save_to_disk`. The train is bigger and ended up in 10 shards, whereas the test split only resulted in 1 split.

Now when trying to load the dataset, an error is raised that not all splits have the same data format:

> ValueError: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('test'): ('json', {})}

This is not expected because both splits are saved as arrow files.

I did some debugging and found that this is the case because the list of data_files includes a `state.json` file. 

Now this means for train split I get 10 ".arrow" and 1 ".json" file. Since datasets picks based on the most common extension this is correctly inferred as "arrow". In the test split, there is 1 .arrow and 1 .json file. Given the function description:

> It picks the module based on the most common file extension.
    In case of a draw ".parquet" is the favorite, and then alphabetical order.

This is not quite true though, because in a tie the extensions are actually based on reverse-alphabetical order:
```
for (ext, _), _ in sorted(extensions_counter.items(), key=sort_key, *reverse=True*):
```

Which thus leads to the module wrongly inferred as "json", whereas it should be "arrow", matching the train split.

I first thought about adding "state.json" in the list of excluded files for the inference: https://github.com/huggingface/datasets/blob/main/src/datasets/load.py#L513. However, I think from digging into the code it looks like the right thing to do is to exclude it in the list of `data_files` to start with, because it is more of a metadata than a data file.