ExpectedMoreSplits error on load_dataset when upgrading to 2.19.0
NONE
Get same error on same datasets too.
2024-04-29T10:33:37Z
https://github.com/huggingface/datasets/issues/6836#issuecomment-2082383839
https://api.github.com/repos/huggingface/datasets/issues/6836
IC_kwDODunzps58Hqff
https://api.github.com/repos/huggingface/datasets/issues/comments/2082383839/reactions
2024-04-29T10:33:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2082383839
https://avatars.githubusercontent.com/u/53668275?v=4
https://api.github.com/users/relic-yuexi/events{/privacy}
https://api.github.com/users/relic-yuexi/followers
https://api.github.com/users/relic-yuexi/following{/other_user}
https://api.github.com/users/relic-yuexi/gists{/gist_id}
https://github.com/relic-yuexi
relic-yuexi
MDQ6VXNlcjUzNjY4Mjc1
https://api.github.com/users/relic-yuexi/orgs
https://api.github.com/users/relic-yuexi/received_events
https://api.github.com/users/relic-yuexi/repos
https://api.github.com/users/relic-yuexi/starred{/owner}{/repo}
https://api.github.com/users/relic-yuexi/subscriptions
User
https://api.github.com/users/relic-yuexi
public
NONE
+1
2024-05-02T14:42:37Z
https://github.com/huggingface/datasets/issues/6836#issuecomment-2090680032
https://api.github.com/repos/huggingface/datasets/issues/6836
IC_kwDODunzps58nT7g
https://api.github.com/repos/huggingface/datasets/issues/comments/2090680032/reactions
2024-05-02T14:42:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2090680032
https://avatars.githubusercontent.com/u/60988468?v=4
https://api.github.com/users/jxmsML/events{/privacy}
https://api.github.com/users/jxmsML/followers
https://api.github.com/users/jxmsML/following{/other_user}
https://api.github.com/users/jxmsML/gists{/gist_id}
https://github.com/jxmsML
jxmsML
MDQ6VXNlcjYwOTg4NDY4
https://api.github.com/users/jxmsML/orgs
https://api.github.com/users/jxmsML/received_events
https://api.github.com/users/jxmsML/repos
https://api.github.com/users/jxmsML/starred{/owner}{/repo}
https://api.github.com/users/jxmsML/subscriptions
User
https://api.github.com/users/jxmsML
public
NONE
same error
2024-05-14T04:08:18Z
https://github.com/huggingface/datasets/issues/6836#issuecomment-2109241437
https://api.github.com/repos/huggingface/datasets/issues/6836
IC_kwDODunzps59uHhd
https://api.github.com/repos/huggingface/datasets/issues/comments/2109241437/reactions
2024-05-14T04:08:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2109241437
https://avatars.githubusercontent.com/u/44458425?v=4
https://api.github.com/users/whwhwwhh/events{/privacy}
https://api.github.com/users/whwhwwhh/followers
https://api.github.com/users/whwhwwhh/following{/other_user}
https://api.github.com/users/whwhwwhh/gists{/gist_id}
https://github.com/whwhwwhh
whwhwwhh
MDQ6VXNlcjQ0NDU4NDI1
https://api.github.com/users/whwhwwhh/orgs
https://api.github.com/users/whwhwwhh/received_events
https://api.github.com/users/whwhwwhh/repos
https://api.github.com/users/whwhwwhh/starred{/owner}{/repo}
https://api.github.com/users/whwhwwhh/subscriptions
User
https://api.github.com/users/whwhwwhh
public
### Describe the bug

Hi there, thanks for the great library! We have been using it a lot in torchtune and it's been a huge help for us. 

Regarding the bug: the same call to `load_dataset` errors with `ExpectedMoreSplits` in 2.19.0 after working fine in 2.18.0. Full details given in the repro below.

### Steps to reproduce the bug

On 2.18.0, things work fine:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.18.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

On 2.19.0, they do not:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.19.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

The stack trace I see from the 2.19.0 version of load_dataset can be seen [here](https://gist.github.com/ebsmothers/f9b1f1949bee7030a8d7bb8a491550d2).

(Maybe unsurprising but) notably if I do not delete the cache first I am able to load the dataset successfully. So based on this I suspect the cause is somewhere in the download logic.

### Expected behavior

Download the dataset successfully :)

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34
- Python version: 3.11.9
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1