Download only split data
CONTRIBUTOR
Hi @capsabogdan! Unfortunately, it's hard to implement because quite often datasets data is being hosted in a single archive for all splits :( So we have to download the whole archive to split it into splits. This is the case for CommonVoice too. 

However, for cases when data is distributed in separate archives ащк different splits I suppose it can (and will) be implemented someday. 


Btw for quick check of the dataset you can use [streaming](https://huggingface.co/docs/datasets/stream):
```python
cv = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)
cv = iter(cv)
print(next(cv))

>> {'client_id': 'a07b17f8234ded5e847443ea6f423cef745cbbc7537fb637d58326000aa751e829a21c4fd0a35fc17fb833aa7e95ebafce5efd19beeb8d843887b85e4eb35f5b',
>>  'path': None,
>>  'audio': {'path': 'cv-corpus-11.0-2022-09-21/en/clips/common_voice_en_100363.mp3',
>>  'array': array([ 0.0000000e+00,  1.1748125e-14,  1.5450088e-14, ...,
>>          1.3011958e-06, -6.3548953e-08, -9.9098514e-08], dtype=float32),
>> ...}

```
2022-11-15T15:55:08Z
https://github.com/huggingface/datasets/issues/5243#issuecomment-1315512887
https://api.github.com/repos/huggingface/datasets/issues/5243
IC_kwDODunzps5OaSI3
https://api.github.com/repos/huggingface/datasets/issues/comments/1315512887/reactions
2022-11-15T15:59:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1315512887
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
NONE
thank you for the answer but am not sure if this will not be helpful, as we
need maybe just 10% of the datasets for some experiment

can we get just a portion of the dataset with stream?


is there really no solution? :(

Am Di., 15. Nov. 2022 um 16:55 Uhr schrieb Polina Kazakova <
***@***.***>:

> Hi @capsabogdan <https://github.com/capsabogdan>! Unfortunately, it's
> hard to implement because quite often datasets data is being hosted in a
> single archive for all splits :( So we have to download the whole archive
> to split it into splits. This is the case for CommonVoice too.
>
> However, for cases when data is distributed in separate archives in
> different splits I suppose it can be implemented someday.
>
> Btw for quick check of the dataset you can use streaming
> <https://huggingface.co/docs/datasets/stream>:
>
> cv = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)cv = iter(cv)print(next(cv))
> >> {'client_id': 'a07b17f8234ded5e847443ea6f423cef745cbbc7537fb637d58326000aa751e829a21c4fd0a35fc17fb833aa7e95ebafce5efd19beeb8d843887b85e4eb35f5b',>>  'path': None,>>  'audio': {'path': 'cv-corpus-11.0-2022-09-21/en/clips/common_voice_en_100363.mp3',>>  'array': array([ 0.0000000e+00,  1.1748125e-14,  1.5450088e-14, ...,>>          1.3011958e-06, -6.3548953e-08, -9.9098514e-08], dtype=float32),>> ...}
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5243#issuecomment-1315512887>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOC3JYRCTH54OBRUJULWIOW6PANCNFSM6AAAAAASAYO2LY>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

2022-11-15T18:24:25Z
https://github.com/huggingface/datasets/issues/5243#issuecomment-1315704321
https://api.github.com/repos/huggingface/datasets/issues/5243
IC_kwDODunzps5ObA4B
https://api.github.com/repos/huggingface/datasets/issues/comments/1315704321/reactions
2022-11-15T18:24:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1315704321
https://avatars.githubusercontent.com/u/48530104?v=4
https://api.github.com/users/capsabogdan/events{/privacy}
https://api.github.com/users/capsabogdan/followers
https://api.github.com/users/capsabogdan/following{/other_user}
https://api.github.com/users/capsabogdan/gists{/gist_id}
https://github.com/capsabogdan
capsabogdan
MDQ6VXNlcjQ4NTMwMTA0
https://api.github.com/users/capsabogdan/orgs
https://api.github.com/users/capsabogdan/received_events
https://api.github.com/users/capsabogdan/repos
https://api.github.com/users/capsabogdan/starred{/owner}{/repo}
https://api.github.com/users/capsabogdan/subscriptions
User
https://api.github.com/users/capsabogdan
public
NONE
maybe it would be nice if you guys ould do some sort of shard before
loading the dataset, so users can download just chunks of data :)

I think this would be very helpful

Am Di., 15. Nov. 2022 um 19:24 Uhr schrieb Bogdan Capsa <
***@***.***>:

> thank you for the answer but am not sure if this will not be helpful, as
> we need maybe just 10% of the datasets for some experiment
>
> can we get just a portion of the dataset with stream?
>
>
> is there really no solution? :(
>
> Am Di., 15. Nov. 2022 um 16:55 Uhr schrieb Polina Kazakova <
> ***@***.***>:
>
>> Hi @capsabogdan <https://github.com/capsabogdan>! Unfortunately, it's
>> hard to implement because quite often datasets data is being hosted in a
>> single archive for all splits :( So we have to download the whole archive
>> to split it into splits. This is the case for CommonVoice too.
>>
>> However, for cases when data is distributed in separate archives in
>> different splits I suppose it can be implemented someday.
>>
>> Btw for quick check of the dataset you can use streaming
>> <https://huggingface.co/docs/datasets/stream>:
>>
>> cv = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)cv = iter(cv)print(next(cv))
>> >> {'client_id': 'a07b17f8234ded5e847443ea6f423cef745cbbc7537fb637d58326000aa751e829a21c4fd0a35fc17fb833aa7e95ebafce5efd19beeb8d843887b85e4eb35f5b',>>  'path': None,>>  'audio': {'path': 'cv-corpus-11.0-2022-09-21/en/clips/common_voice_en_100363.mp3',>>  'array': array([ 0.0000000e+00,  1.1748125e-14,  1.5450088e-14, ...,>>          1.3011958e-06, -6.3548953e-08, -9.9098514e-08], dtype=float32),>> ...}
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/datasets/issues/5243#issuecomment-1315512887>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ALSIFOC3JYRCTH54OBRUJULWIOW6PANCNFSM6AAAAAASAYO2LY>
>> .
>> You are receiving this because you were mentioned.Message ID:
>> ***@***.***>
>>
>

2022-11-15T20:12:24Z
https://github.com/huggingface/datasets/issues/5243#issuecomment-1315812624
https://api.github.com/repos/huggingface/datasets/issues/5243
IC_kwDODunzps5ObbUQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1315812624/reactions
2022-11-15T20:12:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1315812624
https://avatars.githubusercontent.com/u/48530104?v=4
https://api.github.com/users/capsabogdan/events{/privacy}
https://api.github.com/users/capsabogdan/followers
https://api.github.com/users/capsabogdan/following{/other_user}
https://api.github.com/users/capsabogdan/gists{/gist_id}
https://github.com/capsabogdan
capsabogdan
MDQ6VXNlcjQ4NTMwMTA0
https://api.github.com/users/capsabogdan/orgs
https://api.github.com/users/capsabogdan/received_events
https://api.github.com/users/capsabogdan/repos
https://api.github.com/users/capsabogdan/starred{/owner}{/repo}
https://api.github.com/users/capsabogdan/subscriptions
User
https://api.github.com/users/capsabogdan
public
NONE
+1 on this feature request - I am running into the same problem, where I only need the test set for a dataset that has a huge training set
2023-01-05T09:01:07Z
https://github.com/huggingface/datasets/issues/5243#issuecomment-1371946421
https://api.github.com/repos/huggingface/datasets/issues/5243
IC_kwDODunzps5Rxj21
https://api.github.com/repos/huggingface/datasets/issues/comments/1371946421/reactions
2023-01-05T09:01:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1371946421
https://avatars.githubusercontent.com/u/6254922?v=4
https://api.github.com/users/rbawden/events{/privacy}
https://api.github.com/users/rbawden/followers
https://api.github.com/users/rbawden/following{/other_user}
https://api.github.com/users/rbawden/gists{/gist_id}
https://github.com/rbawden
rbawden
MDQ6VXNlcjYyNTQ5MjI=
https://api.github.com/users/rbawden/orgs
https://api.github.com/users/rbawden/received_events
https://api.github.com/users/rbawden/repos
https://api.github.com/users/rbawden/starred{/owner}{/repo}
https://api.github.com/users/rbawden/subscriptions
User
https://api.github.com/users/rbawden
public
NONE
Hey, I'm also interested in that as a feature. I'm having the same problem with Common Voice 13.0. The dataset is super big but I only want the test data to benchmark multilingual models, but I don't have much Terabytes to store all the dataset...
2023-05-02T09:27:51Z
https://github.com/huggingface/datasets/issues/5243#issuecomment-1531161405
https://api.github.com/repos/huggingface/datasets/issues/5243
IC_kwDODunzps5bQ6s9
https://api.github.com/repos/huggingface/datasets/issues/comments/1531161405/reactions
2023-05-02T09:27:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1531161405
https://avatars.githubusercontent.com/u/59816527?v=4
https://api.github.com/users/thomas-ferraz/events{/privacy}
https://api.github.com/users/thomas-ferraz/followers
https://api.github.com/users/thomas-ferraz/following{/other_user}
https://api.github.com/users/thomas-ferraz/gists{/gist_id}
https://github.com/thomas-ferraz
thomas-ferraz
MDQ6VXNlcjU5ODE2NTI3
https://api.github.com/users/thomas-ferraz/orgs
https://api.github.com/users/thomas-ferraz/received_events
https://api.github.com/users/thomas-ferraz/repos
https://api.github.com/users/thomas-ferraz/starred{/owner}{/repo}
https://api.github.com/users/thomas-ferraz/subscriptions
User
https://api.github.com/users/thomas-ferraz
public
NONE
Consider this approach: Download and save individual audio files by streaming each split, then compile a CSV file that contains the file names and corresponding text.

```python3
import os
import shutil
from pathlib import Path

import datasets
import pandas as pd
import soundfile
from datasets import Dataset, concatenate_datasets, load_dataset


dataset = load_dataset("librispeech_asr", 'clean', split="train.100", streaming=True)
dataset = iter(dataset)

download_path = os.path.join(os.getcwd(), 'librispeech', 'clips')
csv_name = os.path.join(os.getcwd(), 'librispeech', 'clean_train_100.csv')

rows = []
for i, row in enumerate(dataset):
    print(i)
    path = os.path.join(download_path, row['audio']['path'])
    soundfile.write(path, row['audio']['array'], row['audio']['sampling_rate'])

    del row['audio']
    rows.append(row)

df = pd.DataFrame(rows)
df.to_csv(csv_name, index=False, header=True)
```
2024-03-06T13:28:15Z
https://github.com/huggingface/datasets/issues/5243#issuecomment-1980874004
https://api.github.com/repos/huggingface/datasets/issues/5243
IC_kwDODunzps52Eb0U
https://api.github.com/repos/huggingface/datasets/issues/comments/1980874004/reactions
2024-03-06T13:28:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1980874004
https://avatars.githubusercontent.com/u/13640533?v=4
https://api.github.com/users/VladimirVincan/events{/privacy}
https://api.github.com/users/VladimirVincan/followers
https://api.github.com/users/VladimirVincan/following{/other_user}
https://api.github.com/users/VladimirVincan/gists{/gist_id}
https://github.com/VladimirVincan
VladimirVincan
MDQ6VXNlcjEzNjQwNTMz
https://api.github.com/users/VladimirVincan/orgs
https://api.github.com/users/VladimirVincan/received_events
https://api.github.com/users/VladimirVincan/repos
https://api.github.com/users/VladimirVincan/starred{/owner}{/repo}
https://api.github.com/users/VladimirVincan/subscriptions
User
https://api.github.com/users/VladimirVincan
public
NONE
Faced this issue as well so wrote a short script that pulls a hub dataset, creates a small sample of it, and pushes the sample data to the hub as a new dataset.

```python
def create_sample_dataset(full_dataset_name, sample_count=100, username="my-username", cache_dir="./dataset"):
    # Create a directory to save the sampled dataset
    os.makedirs(cache_dir, exist_ok=True)

    # Get the dataset name
    dataset_name = full_dataset_name.split("/")[-1]
    dataset_name_sample = f"{dataset_name}-sample-{sample_count}"

    # Load the dataset
    dataset = datasets.load_dataset(full_dataset_name, cache_dir=cache_dir)

    # Sample 100 rows from the training split (or modify for other splits)
    train_sample = dataset["train"].shuffle(seed=42).select(range(sample_count))
    test_sample = dataset["test"].shuffle(seed=42).select(range(sample_count))

    # Push to hub
    train_sample.push_to_hub(dataset_name_sample, split="train")
    print("INFO: Train split pushed to the hub successfully")

    test_sample.push_to_hub(dataset_name_sample, split="test")
    print("INFO: Test split pushed to the hub successfully")
```

Once sampled / pushed you have a smaller version of your dataset in the hub to pull from.

The [full gist is here](https://gist.github.com/neonwatty/2205277501928f5945726e9e90950ae9).
2025-02-25T14:46:38Z
https://github.com/huggingface/datasets/issues/5243#issuecomment-2682237837
https://api.github.com/repos/huggingface/datasets/issues/5243
IC_kwDODunzps6f37ON
https://api.github.com/repos/huggingface/datasets/issues/comments/2682237837/reactions
2025-02-25T14:47:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2682237837
https://avatars.githubusercontent.com/u/16326421?v=4
https://api.github.com/users/neonwatty/events{/privacy}
https://api.github.com/users/neonwatty/followers
https://api.github.com/users/neonwatty/following{/other_user}
https://api.github.com/users/neonwatty/gists{/gist_id}
https://github.com/neonwatty
neonwatty
MDQ6VXNlcjE2MzI2NDIx
https://api.github.com/users/neonwatty/orgs
https://api.github.com/users/neonwatty/received_events
https://api.github.com/users/neonwatty/repos
https://api.github.com/users/neonwatty/starred{/owner}{/repo}
https://api.github.com/users/neonwatty/subscriptions
User
https://api.github.com/users/neonwatty
public
### Feature request

Is it possible to download only the data that I am requesting and not the entire dataset? I run out of disk spaceas it seems to download the entire dataset, instead of only the part needed.

common_voice["test"] = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", 
                                    cache_dir="cache/path...",
                                    use_auth_token=True,
                                    download_config=DownloadConfig(delete_extracted='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')
                                    )


### Motivation

efficiency improvement

### Your contribution

n/a