any possibility to download part of large datasets only?
MEMBER
Not yet but it’s on the short/mid-term roadmap (requested by many indeed).
2021-04-02T10:07:49Z
https://github.com/huggingface/datasets/issues/2161#issuecomment-812465820
https://api.github.com/repos/huggingface/datasets/issues/2161
MDEyOklzc3VlQ29tbWVudDgxMjQ2NTgyMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/812465820/reactions
2021-04-02T10:07:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/812465820
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
oh, great, really awesome feature to have, thank you very much for the great, fabulous work
2021-04-02T13:57:25Z
https://github.com/huggingface/datasets/issues/2161#issuecomment-812541992
https://api.github.com/repos/huggingface/datasets/issues/2161
MDEyOklzc3VlQ29tbWVudDgxMjU0MTk5Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/812541992/reactions
2021-04-02T13:57:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/812541992
https://avatars.githubusercontent.com/u/79165106?v=4
https://api.github.com/users/dorost1234/events{/privacy}
https://api.github.com/users/dorost1234/followers
https://api.github.com/users/dorost1234/following{/other_user}
https://api.github.com/users/dorost1234/gists{/gist_id}
https://github.com/dorost1234
dorost1234
MDQ6VXNlcjc5MTY1MTA2
https://api.github.com/users/dorost1234/orgs
https://api.github.com/users/dorost1234/received_events
https://api.github.com/users/dorost1234/repos
https://api.github.com/users/dorost1234/starred{/owner}{/repo}
https://api.github.com/users/dorost1234/subscriptions
User
https://api.github.com/users/dorost1234
public
MEMBER
We'll work on dataset streaming soon. This should allow you to only load the examples you need ;)
2021-04-07T10:14:36Z
https://github.com/huggingface/datasets/issues/2161#issuecomment-814791922
https://api.github.com/repos/huggingface/datasets/issues/2161
MDEyOklzc3VlQ29tbWVudDgxNDc5MTkyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/814791922/reactions
2021-04-07T10:14:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814791922
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
thanks a lot Quentin, this would be really really a great feature to have

On Wed, Apr 7, 2021 at 12:14 PM Quentin Lhoest ***@***.***>
wrote:

> We'll work on dataset streaming soon. This should allow you to only load
> the examples you need ;)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2161#issuecomment-814791922>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMROD62QAKIJMAKWISTTHQWBVANCNFSM42IUI5JQ>
> .
>

2021-04-07T12:05:45Z
https://github.com/huggingface/datasets/issues/2161#issuecomment-814859673
https://api.github.com/repos/huggingface/datasets/issues/2161
MDEyOklzc3VlQ29tbWVudDgxNDg1OTY3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/814859673/reactions
2021-04-07T12:05:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814859673
https://avatars.githubusercontent.com/u/79165106?v=4
https://api.github.com/users/dorost1234/events{/privacy}
https://api.github.com/users/dorost1234/followers
https://api.github.com/users/dorost1234/following{/other_user}
https://api.github.com/users/dorost1234/gists{/gist_id}
https://github.com/dorost1234
dorost1234
MDQ6VXNlcjc5MTY1MTA2
https://api.github.com/users/dorost1234/orgs
https://api.github.com/users/dorost1234/received_events
https://api.github.com/users/dorost1234/repos
https://api.github.com/users/dorost1234/starred{/owner}{/repo}
https://api.github.com/users/dorost1234/subscriptions
User
https://api.github.com/users/dorost1234
public
NONE
Is streaming completed? On the 1.8.0 docs it is mentioned (https://huggingface.co/docs/datasets/dataset_streaming.html), but when following the example I get the following error:

```
>>> dataset2 = load_dataset("amazon_us_reviews", "Pet_Products_v1_00", split='train', streaming=True)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-1eedab26cff1> in <module>()
----> 1 en_dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)

3 frames
/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    339                 if value is not None:
    340                     if not hasattr(builder_config, key):
--> 341                         raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
    342                     setattr(builder_config, key, value)
    343 

ValueError: BuilderConfig OscarConfig(name='unshuffled_deduplicated_en', version=1.0.0, data_dir=None, data_files=None, description='Unshuffled and deduplicated, English OSCAR dataset') doesn't have a 'streaming' key.
```

UPDATE: Managed to get streaming working by building from source and installing the additional `datasets[streaming]` package:

```
!pip install git+https://github.com/huggingface/datasets.git
!pip install datasets[streaming]
```
2021-07-02T13:21:45Z
https://github.com/huggingface/datasets/issues/2161#issuecomment-872994496
https://api.github.com/repos/huggingface/datasets/issues/2161
MDEyOklzc3VlQ29tbWVudDg3Mjk5NDQ5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/872994496/reactions
2021-07-02T14:23:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/872994496
https://avatars.githubusercontent.com/u/26921364?v=4
https://api.github.com/users/joshbriegal/events{/privacy}
https://api.github.com/users/joshbriegal/followers
https://api.github.com/users/joshbriegal/following{/other_user}
https://api.github.com/users/joshbriegal/gists{/gist_id}
https://github.com/joshbriegal
joshbriegal
MDQ6VXNlcjI2OTIxMzY0
https://api.github.com/users/joshbriegal/orgs
https://api.github.com/users/joshbriegal/received_events
https://api.github.com/users/joshbriegal/repos
https://api.github.com/users/joshbriegal/starred{/owner}{/repo}
https://api.github.com/users/joshbriegal/subscriptions
User
https://api.github.com/users/joshbriegal
public
MEMBER
Hi ! Streaming is available on `master` only right now. We'll make a new release 1.9.0 on Monday :)
2021-07-02T15:19:29Z
https://github.com/huggingface/datasets/issues/2161#issuecomment-873076503
https://api.github.com/repos/huggingface/datasets/issues/2161
MDEyOklzc3VlQ29tbWVudDg3MzA3NjUwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/873076503/reactions
2021-07-02T15:19:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/873076503
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hi
Some of the datasets I need like cc100 are very large, and then I wonder if I can download first X samples of the shuffled/unshuffled data without going through first downloading the whole data then sampling? thanks