Support downloading specific splits in `load_dataset`
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6832). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2024-04-23T12:37:45Z
https://github.com/huggingface/datasets/pull/6832#issuecomment-2072190675
https://api.github.com/repos/huggingface/datasets/issues/6832
IC_kwDODunzps57gx7T
https://api.github.com/repos/huggingface/datasets/issues/comments/2072190675/reactions
2024-04-23T12:37:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2072190675
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
Friendly ping on this! This feature would be really helpful and useful to me (and likely others with limited download speed and storage space!). Thanks so much!
2024-08-19T14:23:23Z
https://github.com/huggingface/datasets/pull/6832#issuecomment-2296706682
https://api.github.com/repos/huggingface/datasets/issues/6832
IC_kwDODunzps6I5PZ6
https://api.github.com/repos/huggingface/datasets/issues/comments/2296706682/reactions
2024-08-19T14:23:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2296706682
https://avatars.githubusercontent.com/u/729312?v=4
https://api.github.com/users/BlackHC/events{/privacy}
https://api.github.com/users/BlackHC/followers
https://api.github.com/users/BlackHC/following{/other_user}
https://api.github.com/users/BlackHC/gists{/gist_id}
https://github.com/BlackHC
BlackHC
MDQ6VXNlcjcyOTMxMg==
https://api.github.com/users/BlackHC/orgs
https://api.github.com/users/BlackHC/received_events
https://api.github.com/users/BlackHC/repos
https://api.github.com/users/BlackHC/starred{/owner}{/repo}
https://api.github.com/users/BlackHC/subscriptions
User
https://api.github.com/users/BlackHC
public
MEMBER
No one is working on this atm afaik :/
2024-08-19T15:12:30Z
https://github.com/huggingface/datasets/pull/6832#issuecomment-2296820164
https://api.github.com/repos/huggingface/datasets/issues/6832
IC_kwDODunzps6I5rHE
https://api.github.com/repos/huggingface/datasets/issues/comments/2296820164/reactions
2024-08-19T15:12:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2296820164
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
No worries! I've patched the ImageNet dataset in: <https://huggingface.co/datasets/ILSVRC/imagenet-1k/blob/refs%2Fpr%2F20/imagenet-1k.py> 

Together with:

```python
dataset = load_dataset(
        "ILSVRC/imagenet-1k",
        split="validation",
        data_files={"val": "data/val_images.tar.gz"},
        revision="refs/pr/20",
        trust_remote_code=True,
        download_config=DownloadConfig(resume_download=True),
        verification_mode=VerificationMode.NO_CHECKS,
    )
```

It only downloads the validation set this way (NO_CHECKS is a bit annoying because I'd rather have md5 checks, but I guess I can't have everything) ^^' The patch is not perfect, but it does the job.


2024-08-19T15:19:37Z
https://github.com/huggingface/datasets/pull/6832#issuecomment-2296835441
https://api.github.com/repos/huggingface/datasets/issues/6832
IC_kwDODunzps6I5u1x
https://api.github.com/repos/huggingface/datasets/issues/comments/2296835441/reactions
2024-08-19T15:19:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2296835441
https://avatars.githubusercontent.com/u/729312?v=4
https://api.github.com/users/BlackHC/events{/privacy}
https://api.github.com/users/BlackHC/followers
https://api.github.com/users/BlackHC/following{/other_user}
https://api.github.com/users/BlackHC/gists{/gist_id}
https://github.com/BlackHC
BlackHC
MDQ6VXNlcjcyOTMxMg==
https://api.github.com/users/BlackHC/orgs
https://api.github.com/users/BlackHC/received_events
https://api.github.com/users/BlackHC/repos
https://api.github.com/users/BlackHC/starred{/owner}{/repo}
https://api.github.com/users/BlackHC/subscriptions
User
https://api.github.com/users/BlackHC
public
CONTRIBUTOR
#self assign 
https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130
2025-07-21T07:49:31Z
https://github.com/huggingface/datasets/pull/6832#issuecomment-3095621653
https://api.github.com/repos/huggingface/datasets/issues/6832
IC_kwDODunzps64g3AV
https://api.github.com/repos/huggingface/datasets/issues/comments/3095621653/reactions
2025-07-21T07:49:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3095621653
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
This PR builds on https://github.com/huggingface/datasets/pull/6639 to support downloading only the specified splits in `load_dataset`. For this to work, a builder's `_split_generators` need to be able to accept the requested splits (as a list) via a `splits` argument to avoid processing the non-requested ones. Also, the builder has to define a `_available_splits` method that lists all the possible `splits` values.

Close https://github.com/huggingface/datasets/issues/4101, close https://github.com/huggingface/datasets/issues/2538 (I'm probably missing some)

Should also make it possible to address https://github.com/huggingface/datasets/issues/6793