[Streaming] Only load requested splits without resolving files for the other splits
MEMBER
This should help fixing this issue: https://github.com/huggingface/datasets/pull/6832
2024-04-29T09:58:18Z
https://github.com/huggingface/datasets/issues/6847#issuecomment-2082321886
https://api.github.com/repos/huggingface/datasets/issues/6847
IC_kwDODunzps58HbXe
https://api.github.com/repos/huggingface/datasets/issues/comments/2082321886/reactions
2024-04-29T09:58:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2082321886
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I'm having a similar issue when using splices:
<img width="947" alt="image" src="https://github.com/huggingface/datasets/assets/28941213/2153faac-e1fe-4b6d-a79b-30b2699407e8">
<img width="823" alt="image" src="https://github.com/huggingface/datasets/assets/28941213/80919eca-eb6c-407d-8070-52642fdcee54">
<img width="914" alt="image" src="https://github.com/huggingface/datasets/assets/28941213/5219c201-e22e-4536-acc3-a922677785ff">


It seems to be downloading, loading, and generating splits using the entire dataset.
2024-05-07T04:43:28Z
https://github.com/huggingface/datasets/issues/6847#issuecomment-2097434909
https://api.github.com/repos/huggingface/datasets/issues/6847
IC_kwDODunzps59BFEd
https://api.github.com/repos/huggingface/datasets/issues/comments/2097434909/reactions
2024-05-07T04:43:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2097434909
https://avatars.githubusercontent.com/u/28941213?v=4
https://api.github.com/users/akaashdash/events{/privacy}
https://api.github.com/users/akaashdash/followers
https://api.github.com/users/akaashdash/following{/other_user}
https://api.github.com/users/akaashdash/gists{/gist_id}
https://github.com/akaashdash
akaashdash
MDQ6VXNlcjI4OTQxMjEz
https://api.github.com/users/akaashdash/orgs
https://api.github.com/users/akaashdash/received_events
https://api.github.com/users/akaashdash/repos
https://api.github.com/users/akaashdash/starred{/owner}{/repo}
https://api.github.com/users/akaashdash/subscriptions
User
https://api.github.com/users/akaashdash
public
e.g. [thangvip](https://huggingface.co/thangvip)/[cosmopedia_vi_math](https://huggingface.co/datasets/thangvip/cosmopedia_vi_math) has 300 splits and it takes a very long time to load only one split.

This is due to `load_dataset()` resolving the files of all the splits even if only one is needed.

In `dataset-viewer` the splits are loaded in different jobs so it results in 300 jobs that resolve 300 splits -> 90k calls to `/paths-info`