Allow downloading just some columns of a dataset
MEMBER
In the general case you can’t always reduce the quantity of data to download, since you can’t parse CSV or JSON data without downloading the whole files right ? ^^ However we could explore this case-by-case I guess
2022-04-06T16:47:29Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-1090484293
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps5A_3hF
https://api.github.com/repos/huggingface/datasets/issues/comments/1090484293/reactions
2022-04-06T16:47:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1090484293
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Actually for csv pandas has `usecols` which allows loading a subset of columns in a more efficient way afaik, but yes, you're right this might be more complex than I thought.
2022-04-07T07:56:26Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-1091235290
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps5BCu3a
https://api.github.com/repos/huggingface/datasets/issues/comments/1091235290/reactions
2022-04-07T07:56:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1091235290
https://avatars.githubusercontent.com/u/7246357?v=4
https://api.github.com/users/osanseviero/events{/privacy}
https://api.github.com/users/osanseviero/followers
https://api.github.com/users/osanseviero/following{/other_user}
https://api.github.com/users/osanseviero/gists{/gist_id}
https://github.com/osanseviero
osanseviero
MDQ6VXNlcjcyNDYzNTc=
https://api.github.com/users/osanseviero/orgs
https://api.github.com/users/osanseviero/received_events
https://api.github.com/users/osanseviero/repos
https://api.github.com/users/osanseviero/starred{/owner}{/repo}
https://api.github.com/users/osanseviero/subscriptions
User
https://api.github.com/users/osanseviero
public
NONE
Bumping the visibility of this :) Is there a recommended way of doing this?
2024-02-20T16:51:04Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-1954633264
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps50gVYw
https://api.github.com/repos/huggingface/datasets/issues/comments/1954633264/reactions
2024-02-20T16:51:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1954633264
https://avatars.githubusercontent.com/u/22435018?v=4
https://api.github.com/users/lukasugar/events{/privacy}
https://api.github.com/users/lukasugar/followers
https://api.github.com/users/lukasugar/following{/other_user}
https://api.github.com/users/lukasugar/gists{/gist_id}
https://github.com/lukasugar
lukasugar
MDQ6VXNlcjIyNDM1MDE4
https://api.github.com/users/lukasugar/orgs
https://api.github.com/users/lukasugar/received_events
https://api.github.com/users/lukasugar/repos
https://api.github.com/users/lukasugar/starred{/owner}{/repo}
https://api.github.com/users/lukasugar/subscriptions
User
https://api.github.com/users/lukasugar
public
MEMBER
Passing `columns=[...]` to `load_dataset()` in streaming mode does work if the dataset is in Parquet format, but for other formats it's either not possible or not implemented
2024-02-21T11:29:33Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-1956450114
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps50nQ9C
https://api.github.com/repos/huggingface/datasets/issues/comments/1956450114/reactions
2024-05-17T09:40:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1956450114
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I tried using the `columns=['bambara']` on this dataset `oza75/bambara-tts` which is in parquet, but it does not work. This feature is really useful because sometimes you don't want to download the whole dataset but just a few columns.
2024-04-07T13:50:57Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2041478075
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps55rnu7
https://api.github.com/repos/huggingface/datasets/issues/comments/2041478075/reactions
2024-04-07T13:50:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2041478075
https://avatars.githubusercontent.com/u/29458652?v=4
https://api.github.com/users/oza75/events{/privacy}
https://api.github.com/users/oza75/followers
https://api.github.com/users/oza75/following{/other_user}
https://api.github.com/users/oza75/gists{/gist_id}
https://github.com/oza75
oza75
MDQ6VXNlcjI5NDU4NjUy
https://api.github.com/users/oza75/orgs
https://api.github.com/users/oza75/received_events
https://api.github.com/users/oza75/repos
https://api.github.com/users/oza75/starred{/owner}{/repo}
https://api.github.com/users/oza75/subscriptions
User
https://api.github.com/users/oza75
public
NONE
It doesn't work for the dataset with `parquet` format. Are we missing something?
2024-05-16T14:16:36Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2115372113
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps5-FgRR
https://api.github.com/repos/huggingface/datasets/issues/comments/2115372113/reactions
2024-05-16T14:16:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2115372113
https://avatars.githubusercontent.com/u/5409901?v=4
https://api.github.com/users/Ravi2712/events{/privacy}
https://api.github.com/users/Ravi2712/followers
https://api.github.com/users/Ravi2712/following{/other_user}
https://api.github.com/users/Ravi2712/gists{/gist_id}
https://github.com/Ravi2712
Ravi2712
MDQ6VXNlcjU0MDk5MDE=
https://api.github.com/users/Ravi2712/orgs
https://api.github.com/users/Ravi2712/received_events
https://api.github.com/users/Ravi2712/repos
https://api.github.com/users/Ravi2712/starred{/owner}{/repo}
https://api.github.com/users/Ravi2712/subscriptions
User
https://api.github.com/users/Ravi2712
public
MEMBER
It only works for `streaming=True`. When not streaming it does download the full files locally before reading the data
2024-05-17T09:41:08Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2117157209
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps5-MUFZ
https://api.github.com/repos/huggingface/datasets/issues/comments/2117157209/reactions
2024-05-17T09:41:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2117157209
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq, I have an audio dataset of 250GB on the huggingface hub in parquet format. I only wanted to load the text column. It is taking a lot of time. It seems like it is downloading audio as well even in streaming mode. 
2024-07-06T01:42:18Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2211571294
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps6D0eZe
https://api.github.com/repos/huggingface/datasets/issues/comments/2211571294/reactions
2024-07-06T01:42:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2211571294
https://avatars.githubusercontent.com/u/34835322?v=4
https://api.github.com/users/kdcyberdude/events{/privacy}
https://api.github.com/users/kdcyberdude/followers
https://api.github.com/users/kdcyberdude/following{/other_user}
https://api.github.com/users/kdcyberdude/gists{/gist_id}
https://github.com/kdcyberdude
kdcyberdude
MDQ6VXNlcjM0ODM1MzIy
https://api.github.com/users/kdcyberdude/orgs
https://api.github.com/users/kdcyberdude/received_events
https://api.github.com/users/kdcyberdude/repos
https://api.github.com/users/kdcyberdude/starred{/owner}{/repo}
https://api.github.com/users/kdcyberdude/subscriptions
User
https://api.github.com/users/kdcyberdude
public
NONE
bump on this
2024-10-18T20:23:13Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2423178348
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps6QbsRs
https://api.github.com/repos/huggingface/datasets/issues/comments/2423178348/reactions
2024-10-18T20:23:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2423178348
https://avatars.githubusercontent.com/u/38530926?v=4
https://api.github.com/users/trojblue/events{/privacy}
https://api.github.com/users/trojblue/followers
https://api.github.com/users/trojblue/following{/other_user}
https://api.github.com/users/trojblue/gists{/gist_id}
https://github.com/trojblue
trojblue
MDQ6VXNlcjM4NTMwOTI2
https://api.github.com/users/trojblue/orgs
https://api.github.com/users/trojblue/received_events
https://api.github.com/users/trojblue/repos
https://api.github.com/users/trojblue/starred{/owner}{/repo}
https://api.github.com/users/trojblue/subscriptions
User
https://api.github.com/users/trojblue
public
NONE
Something like this worked for me:
```py
ds = load_dataset(
    "parler-tts/libritts_r_filtered",
    "clean",
    streaming=True,
    columns=['text_normalized']
)
```
2025-02-03T10:07:27Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2630495282
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps6cyiwy
https://api.github.com/repos/huggingface/datasets/issues/comments/2630495282/reactions
2025-02-03T10:07:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2630495282
https://avatars.githubusercontent.com/u/26504141?v=4
https://api.github.com/users/xenova/events{/privacy}
https://api.github.com/users/xenova/followers
https://api.github.com/users/xenova/following{/other_user}
https://api.github.com/users/xenova/gists{/gist_id}
https://github.com/xenova
xenova
MDQ6VXNlcjI2NTA0MTQx
https://api.github.com/users/xenova/orgs
https://api.github.com/users/xenova/received_events
https://api.github.com/users/xenova/repos
https://api.github.com/users/xenova/starred{/owner}{/repo}
https://api.github.com/users/xenova/subscriptions
User
https://api.github.com/users/xenova
public
NONE
I want to get all the captions from this dataset https://huggingface.co/datasets/gmongaras/Imagenet21K_Recaption
and annoyingly you need 2x the disk space listed (3.8tb downloaded, and then it needs another 3.8 tb when it does the "generating train split") to download and load the dataset using load_dataset. So that made it unfeasible for me, I even wasted a few hours trying to do it on a rented instance with 5tb of disk space (which i thought would be more than enough, but isnt because of the 2x disk space thing)

Doing it in streaming mode is way too slow, going through all the rows will take ages. Please make a simpler solution to just get a few columns
2025-02-14T10:45:11Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2658957449
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps6efHiJ
https://api.github.com/repos/huggingface/datasets/issues/comments/2658957449/reactions
2025-02-14T10:46:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2658957449
https://avatars.githubusercontent.com/u/46050679?v=4
https://api.github.com/users/SwayStar123/events{/privacy}
https://api.github.com/users/SwayStar123/followers
https://api.github.com/users/SwayStar123/following{/other_user}
https://api.github.com/users/SwayStar123/gists{/gist_id}
https://github.com/SwayStar123
SwayStar123
MDQ6VXNlcjQ2MDUwNjc5
https://api.github.com/users/SwayStar123/orgs
https://api.github.com/users/SwayStar123/received_events
https://api.github.com/users/SwayStar123/repos
https://api.github.com/users/SwayStar123/starred{/owner}{/repo}
https://api.github.com/users/SwayStar123/subscriptions
User
https://api.github.com/users/SwayStar123
public
MEMBER
You stream the dataset and speed it up with multiple workers, can you try this ?

```python
from datasets import load_dataset
from torch.utils.data import DataLoader

ds = load_dataset("gmongaras/Imagenet21K_Recaption", split="train", streaming=True, columns=["recaption"])
dl = DataLoader(ds, num_workers=8)

for example in dl:
    ...
```

You can also be interested in checkpointing/resuming the streaming (e.g. in case of connection error), check this [guide](https://huggingface.co/docs/datasets/v3.3.0/en/use_with_pytorch#checkpoint-and-resume) for more details

Alternatively feel free to try with other tools that can load Parquet and select columns, like DuckDB / Polars / Spark (see the list of integrated libraries [here](https://huggingface.co/docs/hub/datasets-libraries)).
2025-02-14T14:57:19Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2659544262
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps6ehWzG
https://api.github.com/repos/huggingface/datasets/issues/comments/2659544262/reactions
2025-02-14T14:59:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2659544262
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> You stream the dataset and speed it up with multiple workers, can you try this ?
> 
> from datasets import load_dataset
> from torch.utils.data import DataLoader
> 
> ds = load_dataset("gmongaras/Imagenet21K_Recaption", split="train", streaming=True, columns=["recaption"])
> dl = DataLoader(ds, num_workers=8)
> 
> for example in dl:
>     ...
> You can also be interested in checkpointing/resuming the streaming (e.g. in case of connection error), check this [guide](https://huggingface.co/docs/datasets/v3.3.0/en/use_with_pytorch#checkpoint-and-resume) for more details
> 
> Alternatively feel free to try with other tools that can load Parquet and select columns, like DuckDB / Polars / Spark (see the list of integrated libraries [here](https://huggingface.co/docs/hub/datasets-libraries)).

the dataloader is faster but not by much. I get 15 its/s. Will take like 270 hours at this pace.
2025-02-14T19:09:48Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2660076972
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps6ejY2s
https://api.github.com/repos/huggingface/datasets/issues/comments/2660076972/reactions
2025-02-14T19:09:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2660076972
https://avatars.githubusercontent.com/u/46050679?v=4
https://api.github.com/users/SwayStar123/events{/privacy}
https://api.github.com/users/SwayStar123/followers
https://api.github.com/users/SwayStar123/following{/other_user}
https://api.github.com/users/SwayStar123/gists{/gist_id}
https://github.com/SwayStar123
SwayStar123
MDQ6VXNlcjQ2MDUwNjc5
https://api.github.com/users/SwayStar123/orgs
https://api.github.com/users/SwayStar123/received_events
https://api.github.com/users/SwayStar123/repos
https://api.github.com/users/SwayStar123/starred{/owner}{/repo}
https://api.github.com/users/SwayStar123/subscriptions
User
https://api.github.com/users/SwayStar123
public
MEMBER
Can you check your connection ? I'm at 1,000+ it/s on colab and also on my (consumer) internet connection.

You can also try to update `datasets`, `pyarrow`, `fsspec` and `huggingface_hub`
2025-02-17T15:10:53Z
https://github.com/huggingface/datasets/issues/4114#issuecomment-2663401024
https://api.github.com/repos/huggingface/datasets/issues/4114
IC_kwDODunzps6ewEZA
https://api.github.com/repos/huggingface/datasets/issues/comments/2663401024/reactions
2025-02-17T15:10:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2663401024
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
**Is your feature request related to a problem? Please describe.**
Some people are interested in doing label analysis of a CV dataset without downloading all the images. Downloading the whole dataset does not always makes sense for this kind of use case

**Describe the solution you'd like**
Be able to just download some columns of a dataset, such as doing
```python
load_dataset("huggan/wikiart",columns=["artist", "genre"])
```

Although this might make things a bit complicated in terms of local caching of datasets.