Loading partial dataset when debugging
MEMBER
Hi ! `load_dataset` downloads the full dataset once and caches it, so that subsequent calls to `load_dataset` just reloads the dataset from your disk.
Then when you specify a `split` in `load_dataset`, it will just load the requested split from the disk. If your specified split is a sliced split (e.g. `"train[:10]"`), then it will load the 10 first rows of the train split that you have on disk.

Therefore, as long as you don't delete your cache, all your calls to `load_dataset` will be very fast. Except the first call that downloads the dataset of course ^^
2021-06-23T13:35:40Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-866842054
https://api.github.com/repos/huggingface/datasets/issues/2538
MDEyOklzc3VlQ29tbWVudDg2Njg0MjA1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/866842054/reactions
2021-06-23T13:35:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/866842054
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
That’s a use case for the new streaming feature, no?
2021-06-23T16:00:54Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-866965568
https://api.github.com/repos/huggingface/datasets/issues/2538
MDEyOklzc3VlQ29tbWVudDg2Njk2NTU2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/866965568/reactions
2021-06-23T16:00:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/866965568
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Hi @reachtarunhere.

Besides the above insights provided by @lhoestq and @thomwolf, there is also a Dataset feature in progress (I plan to finish it this week): #2249, which will allow you, when calling `load_dataset`, to pass the option to download/preprocess/cache only some specific split(s), which will definitely speed up your workflow.

If this feature is interesting for you, I can ping you once it will be merged into the master branch.
2021-06-24T05:51:44Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-867355166
https://api.github.com/repos/huggingface/datasets/issues/2538
MDEyOklzc3VlQ29tbWVudDg2NzM1NTE2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/867355166/reactions
2021-06-24T05:51:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/867355166
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Thanks all for responding.

Hey @albertvillanova 

Thanks. Yes, I would be interested.

@lhoestq I think even if a small split is specified it loads up the full dataset from the disk (please correct me if this is not the case). Because it does seem to be slow to me even on subsequent calls. There is no repeated downloading so it seems that the cache is working.

I am not aware of the streaming feature @thomwolf mentioned. So I might need to read up on it.
2021-06-24T07:12:07Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-867398154
https://api.github.com/repos/huggingface/datasets/issues/2538
MDEyOklzc3VlQ29tbWVudDg2NzM5ODE1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/867398154/reactions
2021-06-24T07:12:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/867398154
https://avatars.githubusercontent.com/u/9061913?v=4
https://api.github.com/users/reachtarunhere/events{/privacy}
https://api.github.com/users/reachtarunhere/followers
https://api.github.com/users/reachtarunhere/following{/other_user}
https://api.github.com/users/reachtarunhere/gists{/gist_id}
https://github.com/reachtarunhere
reachtarunhere
MDQ6VXNlcjkwNjE5MTM=
https://api.github.com/users/reachtarunhere/orgs
https://api.github.com/users/reachtarunhere/received_events
https://api.github.com/users/reachtarunhere/repos
https://api.github.com/users/reachtarunhere/starred{/owner}{/repo}
https://api.github.com/users/reachtarunhere/subscriptions
User
https://api.github.com/users/reachtarunhere
public
CONTRIBUTOR
@reshinthadithyan  I use the .select function to have a fraction of indices.
2021-07-29T14:10:14Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-889177637
https://api.github.com/repos/huggingface/datasets/issues/2538
IC_kwDODunzps40_8Yl
https://api.github.com/repos/huggingface/datasets/issues/comments/889177637/reactions
2021-07-29T14:10:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/889177637
https://avatars.githubusercontent.com/u/22492839?v=4
https://api.github.com/users/PierreColombo/events{/privacy}
https://api.github.com/users/PierreColombo/followers
https://api.github.com/users/PierreColombo/following{/other_user}
https://api.github.com/users/PierreColombo/gists{/gist_id}
https://github.com/PierreColombo
PierreColombo
MDQ6VXNlcjIyNDkyODM5
https://api.github.com/users/PierreColombo/orgs
https://api.github.com/users/PierreColombo/received_events
https://api.github.com/users/PierreColombo/repos
https://api.github.com/users/PierreColombo/starred{/owner}{/repo}
https://api.github.com/users/PierreColombo/subscriptions
User
https://api.github.com/users/PierreColombo
public
NONE
If I want to create a dataset, containing only the 10 elements of a given dataset (slice it), how do I do that?
2023-04-10T06:50:37Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-1501466785
https://api.github.com/repos/huggingface/datasets/issues/2538
IC_kwDODunzps5ZfpCh
https://api.github.com/repos/huggingface/datasets/issues/comments/1501466785/reactions
2023-04-10T06:50:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1501466785
https://avatars.githubusercontent.com/u/845175?v=4
https://api.github.com/users/ndvbd/events{/privacy}
https://api.github.com/users/ndvbd/followers
https://api.github.com/users/ndvbd/following{/other_user}
https://api.github.com/users/ndvbd/gists{/gist_id}
https://github.com/ndvbd
ndvbd
MDQ6VXNlcjg0NTE3NQ==
https://api.github.com/users/ndvbd/orgs
https://api.github.com/users/ndvbd/received_events
https://api.github.com/users/ndvbd/repos
https://api.github.com/users/ndvbd/starred{/owner}{/repo}
https://api.github.com/users/ndvbd/subscriptions
User
https://api.github.com/users/ndvbd
public
MEMBER
```python 
small_ds = ds.select(range(10))
```
2023-04-10T09:52:26Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-1501621512
https://api.github.com/repos/huggingface/datasets/issues/2538
IC_kwDODunzps5ZgO0I
https://api.github.com/repos/huggingface/datasets/issues/comments/1501621512/reactions
2023-04-10T09:52:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1501621512
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE


> ```python
> small_ds = ds.select(range(10))
> ```

Thanks, but this doesn't help me to save time during initial loading, right?
2023-04-10T10:50:57Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-1501671843
https://api.github.com/repos/huggingface/datasets/issues/2538
IC_kwDODunzps5ZgbGj
https://api.github.com/repos/huggingface/datasets/issues/comments/1501671843/reactions
2023-04-10T10:51:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1501671843
https://avatars.githubusercontent.com/u/845175?v=4
https://api.github.com/users/ndvbd/events{/privacy}
https://api.github.com/users/ndvbd/followers
https://api.github.com/users/ndvbd/following{/other_user}
https://api.github.com/users/ndvbd/gists{/gist_id}
https://github.com/ndvbd
ndvbd
MDQ6VXNlcjg0NTE3NQ==
https://api.github.com/users/ndvbd/orgs
https://api.github.com/users/ndvbd/received_events
https://api.github.com/users/ndvbd/repos
https://api.github.com/users/ndvbd/starred{/owner}{/repo}
https://api.github.com/users/ndvbd/subscriptions
User
https://api.github.com/users/ndvbd
public
MEMBER
Indeed by default load_dataset would download and prepare everything as Arrow files. And passing `split=train[:10]` memory maps only the beginning of the full dataset that has been prepared on disk.

If you don't want to download everything, you can use streaming : 
```python 
ids = load_dataset(..., streaming=True)
first_samples = list(ids["train"].take(10))
```

To get a Dataset you can use 
```python 
ds = Dataset.from_generator(ids.take(10).__iter__)
```

edit: fixed small bug
2023-04-12T09:50:16Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-1504985607
https://api.github.com/repos/huggingface/datasets/issues/2538
IC_kwDODunzps5ZtEIH
https://api.github.com/repos/huggingface/datasets/issues/comments/1504985607/reactions
2023-04-19T11:05:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1504985607
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks @lhoestq, but I don't think it is 100% accurate, as it doesn't keep the dataset structure exactly the same.
To load the full dataset, I do:
```
data = load_dataset("json", data_files="a.json")
train_data = data["train"].shuffle()
```

But when I am changing it as per your instructions: 
```
ids = load_dataset("json", data_files="a.json", streaming=True)
data = Dataset.from_generator(ids["train"].take(1).__iter__)
train_data = data["train"].shuffle()
```
It throws KeyError.
I need a simple way, like you suggested, to have a subset of a Dataset, which exactly the same attributes.

2023-04-13T09:14:30Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-1506626057
https://api.github.com/repos/huggingface/datasets/issues/2538
IC_kwDODunzps5ZzUoJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1506626057/reactions
2023-04-13T09:14:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1506626057
https://avatars.githubusercontent.com/u/845175?v=4
https://api.github.com/users/ndvbd/events{/privacy}
https://api.github.com/users/ndvbd/followers
https://api.github.com/users/ndvbd/following{/other_user}
https://api.github.com/users/ndvbd/gists{/gist_id}
https://github.com/ndvbd
ndvbd
MDQ6VXNlcjg0NTE3NQ==
https://api.github.com/users/ndvbd/orgs
https://api.github.com/users/ndvbd/received_events
https://api.github.com/users/ndvbd/repos
https://api.github.com/users/ndvbd/starred{/owner}{/repo}
https://api.github.com/users/ndvbd/subscriptions
User
https://api.github.com/users/ndvbd
public
MEMBER
Whoops I fixed my code sorry
```diff
- ds = Dataset.from_generator(ids["train"].take(10).__iter__)
+ ds = Dataset.from_generator(ids.take(10).__iter__)
```

in your case that means running
```python
train_data = data.shuffle()
```

without `["train"]`
2023-04-19T11:04:45Z
https://github.com/huggingface/datasets/issues/2538#issuecomment-1514542591
https://api.github.com/repos/huggingface/datasets/issues/2538
IC_kwDODunzps5aRhX_
https://api.github.com/repos/huggingface/datasets/issues/comments/1514542591/reactions
2023-04-19T11:05:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1514542591
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I am using PyTorch Lightning along with datasets (thanks for so many datasets already prepared and the great splits). 

Every time I execute load_dataset  for the imdb dataset it takes some time even if I specify a split involving very few samples. I guess this due to hashing as per the other issues.

Is there a way to only load part of the dataset on load_dataset? This would really speed up my workflow.
Something like a debug mode would really help. Thanks!