Dataset librispeech_asr fails to load
NONE
@patrickvonplaten Hi! I saw that you prepared this? :)
2022-04-19T09:14:53Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102361556
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BtLPU
https://api.github.com/repos/huggingface/datasets/issues/comments/1102361556/reactions
2022-04-19T09:14:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102361556
https://avatars.githubusercontent.com/u/59132?v=4
https://api.github.com/users/albertz/events{/privacy}
https://api.github.com/users/albertz/followers
https://api.github.com/users/albertz/following{/other_user}
https://api.github.com/users/albertz/gists{/gist_id}
https://github.com/albertz
albertz
MDQ6VXNlcjU5MTMy
https://api.github.com/users/albertz/orgs
https://api.github.com/users/albertz/received_events
https://api.github.com/users/albertz/repos
https://api.github.com/users/albertz/starred{/owner}{/repo}
https://api.github.com/users/albertz/subscriptions
User
https://api.github.com/users/albertz
public
NONE
Another thing, but maybe this should be a separate issue: As I see from the code, it would try to use up to 16 simultaneous downloads? This is problematic for Librispeech or anything on OpenSLR. On [the homepage](https://www.openslr.org/), it says:

> If you want to download things from this site, please download them one at a time, and please don't use any fancy software-- just download things from your browser or use 'wget'. We have a firewall rule to drop connections from hosts with more than 5 simultaneous connections, and certain types of download software may activate this rule.

Related: https://github.com/tensorflow/datasets/issues/3885
2022-04-19T09:18:23Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102366621
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BtMed
https://api.github.com/repos/huggingface/datasets/issues/comments/1102366621/reactions
2022-04-19T09:18:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102366621
https://avatars.githubusercontent.com/u/59132?v=4
https://api.github.com/users/albertz/events{/privacy}
https://api.github.com/users/albertz/followers
https://api.github.com/users/albertz/following{/other_user}
https://api.github.com/users/albertz/gists{/gist_id}
https://github.com/albertz
albertz
MDQ6VXNlcjU5MTMy
https://api.github.com/users/albertz/orgs
https://api.github.com/users/albertz/received_events
https://api.github.com/users/albertz/repos
https://api.github.com/users/albertz/starred{/owner}{/repo}
https://api.github.com/users/albertz/subscriptions
User
https://api.github.com/users/albertz
public
CONTRIBUTOR
Hey @albertz,

Nice to see you here! It's been a while ;-) 
2022-04-19T09:24:17Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102375452
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BtOoc
https://api.github.com/repos/huggingface/datasets/issues/comments/1102375452/reactions
2022-04-19T09:24:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102375452
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
Sorry maybe the docs haven't been super clear here. By `split` we mean one of `train.500`, `train.360`, `train.100`, `validation`, `test`. For Librispeech, you'll have to specific a config (either `other` or `clean`) though:

```py
datasets.load_dataset("librispeech_asr", "clean")
```

should work and give you all splits (being "train", "test", ...) for the clean config of the dataset.

2022-04-19T09:27:46Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102380412
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BtP18
https://api.github.com/repos/huggingface/datasets/issues/comments/1102380412/reactions
2022-04-19T09:27:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102380412
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
If you need both `"clean"` and `"other"` I think you'll have to do concatenate them as follows: 

```py
from datasets import concatenate_datasets, load_dataset

other = load_dataset("librispeech_asr", "other")
clean = load_dataset("librispeech_asr", "clean")

librispeech = concatenate_datasets([other, clean])
```

See https://huggingface.co/docs/datasets/v2.1.0/en/process#concatenate
2022-04-19T09:30:03Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102383717
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BtQpl
https://api.github.com/repos/huggingface/datasets/issues/comments/1102383717/reactions
2022-04-19T09:30:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102383717
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
Downloading one split would be:

```py
from datasets import load_dataset

other = load_dataset("librispeech_asr", "other", split="train.500")
```



2022-04-19T09:30:48Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102384597
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BtQ3V
https://api.github.com/repos/huggingface/datasets/issues/comments/1102384597/reactions
2022-04-19T09:30:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102384597
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
cc @lhoestq FYI maybe the docs can be improved here
2022-04-19T09:31:06Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102384969
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BtQ9J
https://api.github.com/repos/huggingface/datasets/issues/comments/1102384969/reactions
2022-04-19T09:31:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102384969
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
NONE
Ah thanks. But wouldn't it be easier/nicer (and more canonical) to just make it in a way that simply `load_dataset("librispeech_asr")` works?
2022-04-19T09:31:20Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102385237
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BtRBV
https://api.github.com/repos/huggingface/datasets/issues/comments/1102385237/reactions
2022-04-19T09:31:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102385237
https://avatars.githubusercontent.com/u/59132?v=4
https://api.github.com/users/albertz/events{/privacy}
https://api.github.com/users/albertz/followers
https://api.github.com/users/albertz/following{/other_user}
https://api.github.com/users/albertz/gists{/gist_id}
https://github.com/albertz
albertz
MDQ6VXNlcjU5MTMy
https://api.github.com/users/albertz/orgs
https://api.github.com/users/albertz/received_events
https://api.github.com/users/albertz/repos
https://api.github.com/users/albertz/starred{/owner}{/repo}
https://api.github.com/users/albertz/subscriptions
User
https://api.github.com/users/albertz
public
CONTRIBUTOR
Pinging @lhoestq here, think this could make sense! Not sure however how the dictionary would then look like
2022-04-19T09:50:40Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102414863
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BtYQP
https://api.github.com/repos/huggingface/datasets/issues/comments/1102414863/reactions
2022-04-19T09:50:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102414863
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
Would it make sense to have `clean` as the default config ?

Also I think `load_dataset("librispeech_asr")` should have raised you an error that says that you need to specify a config

I also opened a PR to improve the doc: https://github.com/huggingface/datasets/pull/4183
2022-04-19T14:23:05Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102719502
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BuioO
https://api.github.com/repos/huggingface/datasets/issues/comments/1102719502/reactions
2022-04-19T14:27:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102719502
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Would it make sense to have `clean` as the default config ?

I think a user would expect that the default would give you the full dataset.

> Also I think `load_dataset("librispeech_asr")` should have raised you an error that says that you need to specify a config

It does raise an error, but this error confused me because I did not understand why I needed a config, or why I could not simply download the whole dataset, which is what people usually do with Librispeech.

2022-04-19T14:34:38Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102732614
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5Bul1G
https://api.github.com/repos/huggingface/datasets/issues/comments/1102732614/reactions
2022-04-19T14:34:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102732614
https://avatars.githubusercontent.com/u/59132?v=4
https://api.github.com/users/albertz/events{/privacy}
https://api.github.com/users/albertz/followers
https://api.github.com/users/albertz/following{/other_user}
https://api.github.com/users/albertz/gists{/gist_id}
https://github.com/albertz
albertz
MDQ6VXNlcjU5MTMy
https://api.github.com/users/albertz/orgs
https://api.github.com/users/albertz/received_events
https://api.github.com/users/albertz/repos
https://api.github.com/users/albertz/starred{/owner}{/repo}
https://api.github.com/users/albertz/subscriptions
User
https://api.github.com/users/albertz
public
CONTRIBUTOR
+1 for @albertz. Also think lots of people download the whole dataset (`"clean"` + `"other"`) for Librispeech.

Think there are also some people though who:
- a) Don't have the memory to store the whole dataset
- b) Just want to evaluate on one of the two configs
2022-04-19T14:37:03Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102735271
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5Bumen
https://api.github.com/repos/huggingface/datasets/issues/comments/1102735271/reactions
2022-04-19T14:37:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102735271
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
Ok ! Adding the "all" configuration would do the job then, thanks ! In the "all" configuration we can merge all the train.xxx splits into one "train" split, or keep them separate depending on what's the most practical to use (probably put everything in "train" no ?)
2022-04-19T15:15:58Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102778290
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5Buw-y
https://api.github.com/repos/huggingface/datasets/issues/comments/1102778290/reactions
2022-04-19T15:15:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102778290
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I'm not too familiar with how to work with HuggingFace datasets, but people often do some curriculum learning scheme, where they start with train.100, later go over to train.100 + train.360, and then later use the whole train (960h). It would be good if this is easily possible.

2022-04-19T15:37:30Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102801656
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5Bu2r4
https://api.github.com/repos/huggingface/datasets/issues/comments/1102801656/reactions
2022-04-19T15:37:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102801656
https://avatars.githubusercontent.com/u/59132?v=4
https://api.github.com/users/albertz/events{/privacy}
https://api.github.com/users/albertz/followers
https://api.github.com/users/albertz/following{/other_user}
https://api.github.com/users/albertz/gists{/gist_id}
https://github.com/albertz
albertz
MDQ6VXNlcjU5MTMy
https://api.github.com/users/albertz/orgs
https://api.github.com/users/albertz/received_events
https://api.github.com/users/albertz/repos
https://api.github.com/users/albertz/starred{/owner}{/repo}
https://api.github.com/users/albertz/subscriptions
User
https://api.github.com/users/albertz
public
CONTRIBUTOR
Hey @albertz, 

opened a PR here. Think by adding the "subdataset" class to each split "train", "dev", "other" as shown here: https://github.com/huggingface/datasets/pull/4184/files#r853272727 it should be easily possible (e.g. with the filter function https://huggingface.co/docs/datasets/v2.1.0/en/package_reference/main_classes#datasets.Dataset.filter )
2022-04-19T16:34:41Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102858193
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BvEfR
https://api.github.com/repos/huggingface/datasets/issues/comments/1102858193/reactions
2022-04-19T16:34:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102858193
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
But also since everything is cached one could also just do:

```python
load_dataset("librispeech", "clean", "train.100")
load_dataset("librispeech", "clean", "train.100+train.360")
load_dataset("librispeech" "all", "train") 
```
2022-04-19T16:36:26Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1102859908
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5BvE6E
https://api.github.com/repos/huggingface/datasets/issues/comments/1102859908/reactions
2022-04-19T16:36:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102859908
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
NONE
Hi @patrickvonplaten ,

load_dataset("librispeech_asr", "clean", "train.100") actually downloads the whole dataset and not the 100 hr split, is this a bug?
2022-05-25T17:11:55Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1137577098
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5DzgyK
https://api.github.com/repos/huggingface/datasets/issues/comments/1137577098/reactions
2022-05-25T17:11:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1137577098
https://avatars.githubusercontent.com/u/104358577?v=4
https://api.github.com/users/SreyanG-NVIDIA/events{/privacy}
https://api.github.com/users/SreyanG-NVIDIA/followers
https://api.github.com/users/SreyanG-NVIDIA/following{/other_user}
https://api.github.com/users/SreyanG-NVIDIA/gists{/gist_id}
https://github.com/SreyanG-NVIDIA
SreyanG-NVIDIA
U_kgDOBjhisQ
https://api.github.com/users/SreyanG-NVIDIA/orgs
https://api.github.com/users/SreyanG-NVIDIA/received_events
https://api.github.com/users/SreyanG-NVIDIA/repos
https://api.github.com/users/SreyanG-NVIDIA/starred{/owner}{/repo}
https://api.github.com/users/SreyanG-NVIDIA/subscriptions
User
https://api.github.com/users/SreyanG-NVIDIA
public
CONTRIBUTOR
Hmm, I don't really see how that's possible: https://github.com/huggingface/datasets/blob/d22e39a0693d4be7410cf9a5d41fd5aac22be3cc/datasets/librispeech_asr/librispeech_asr.py#L51

Note that all datasets related to `"clean"` are downloaded, but only `"train.100"` should be used. 

cc @lhoestq @albertvillanova @mariosasko can we do anything against download dataset links that are not related to the "split" that one actually needs. E.g. why should the split `"train.360"` be downloaded if for the user executes the above command:

```py
load_dataset("librispeech_asr", "clean", "train.100")
```
2022-05-25T19:36:30Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1137773737
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5D0Qyp
https://api.github.com/repos/huggingface/datasets/issues/comments/1137773737/reactions
2022-05-25T19:36:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1137773737
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
COLLABORATOR
@patrickvonplaten This problem is a bit harder than it may seem, and it has to do with how our scripts are structured - `_split_generators` downloads data for a split before its definition. There was an attempt to fix this in https://github.com/huggingface/datasets/pull/2249, but it wasn't flexible enough. Luckily, I have a plan of attack, and this issue is on our short-term roadmap, so I'll work on it soon.

In the meantime, one can use streaming or manually download a dataset script, remove unwanted splits and load a dataset via `load_dataset`.
2022-05-26T16:59:03Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1138796742
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5D4KjG
https://api.github.com/repos/huggingface/datasets/issues/comments/1138796742/reactions
2022-05-26T16:59:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1138796742
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
> load_dataset("librispeech_asr", "clean", "train.100") actually downloads the whole dataset and not the 100 hr split, is this a bug?

Since this bug is still there and google led me here when I was searching for a solution, I am writing down how to quickly fix it (as suggested by @mariosasko) for whoever else is not familiar with how the HF Hub works.

Download the [librispeech_asr.py](https://huggingface.co/datasets/librispeech_asr/blob/main/librispeech_asr.py) script and remove the unwanted splits both from the [`_DL_URLS` dictionary](https://huggingface.co/datasets/librispeech_asr/blob/main/librispeech_asr.py#L47-L68) and from the [`_split_generators` function](https://huggingface.co/datasets/librispeech_asr/blob/main/librispeech_asr.py#L121-L241).
[Here ](https://huggingface.co/datasets/andreagasparini/librispeech_test_only) I made an example with only the test sets.

Then either save the script locally and load the dataset via 
```python
load_dataset("${local_path}/librispeech_asr.py")
```

or [create a new dataset repo on the hub](https://huggingface.co/new-dataset) named "librispeech_asr" and upload the script there, then you can just run
```python
load_dataset("${hugging_face_username}/librispeech_asr")
```
2022-07-07T09:09:33Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1177289027
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5GLAFD
https://api.github.com/repos/huggingface/datasets/issues/comments/1177289027/reactions
2022-07-07T09:10:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1177289027
https://avatars.githubusercontent.com/u/36965464?v=4
https://api.github.com/users/andrea-gasparini/events{/privacy}
https://api.github.com/users/andrea-gasparini/followers
https://api.github.com/users/andrea-gasparini/following{/other_user}
https://api.github.com/users/andrea-gasparini/gists{/gist_id}
https://github.com/andrea-gasparini
andrea-gasparini
MDQ6VXNlcjM2OTY1NDY0
https://api.github.com/users/andrea-gasparini/orgs
https://api.github.com/users/andrea-gasparini/received_events
https://api.github.com/users/andrea-gasparini/repos
https://api.github.com/users/andrea-gasparini/starred{/owner}{/repo}
https://api.github.com/users/andrea-gasparini/subscriptions
User
https://api.github.com/users/andrea-gasparini
public
MEMBER
Fixed by https://github.com/huggingface/datasets/pull/4184
2022-07-27T16:10:00Z
https://github.com/huggingface/datasets/issues/4179#issuecomment-1196960813
https://api.github.com/repos/huggingface/datasets/issues/4179
IC_kwDODunzps5HWCwt
https://api.github.com/repos/huggingface/datasets/issues/comments/1196960813/reactions
2022-07-27T16:10:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1196960813
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug
The dataset librispeech_asr (standard Librispeech) fails to load.

## Steps to reproduce the bug
```python
datasets.load_dataset("librispeech_asr")
```

## Expected results
It should download and prepare the whole dataset (all subsets).

In [the doc](https://huggingface.co/datasets/librispeech_asr), it says it has two configurations (clean and other).
However, the dataset doc says that not specifying `split` should just load the whole dataset, which is what I want.

Also, in case of this specific dataset, this is also the standard what the community uses. When you look at any publications with results on Librispeech, they always use the whole train dataset for training.

## Actual results
```
...
  File "/home/az/.cache/huggingface/modules/datasets_modules/datasets/librispeech_asr/1f4602f6b5fed8d3ab3e3382783173f2e12d9877e98775e34d7780881175096c/librispeech_asr.py", line 119, in LibrispeechASR._split_generators
    line: archive_path = dl_manager.download(_DL_URLS[self.config.name])
    locals:
      archive_path = <not found>
      dl_manager = <local> <datasets.utils.download_manager.DownloadManager object at 0x7fc07b426160>
      dl_manager.download = <local> <bound method DownloadManager.download of <datasets.utils.download_manager.DownloadManager object at 0x7fc07b426160>>
      _DL_URLS = <global> {'clean': {'dev': 'http://www.openslr.org/resources/12/dev-clean.tar.gz', 'test': 'http://www.openslr.org/resources/12/test-clean.tar.gz', 'train.100': 'http://www.openslr.org/resources/12/train-clean-100.tar.gz', 'train.360': 'http://www.openslr.org/resources/12/train-clean-360.tar.gz'}, 'other'...
      self = <local> <datasets_modules.datasets.librispeech_asr.1f4602f6b5fed8d3ab3e3382783173f2e12d9877e98775e34d7780881175096c.librispeech_asr.LibrispeechASR object at 0x7fc12a633310>
      self.config = <local> BuilderConfig(name='default', version=0.0.0, data_dir='/home/az/i6/setups/2022-03-20--sis/work/i6_core/datasets/huggingface/DownloadAndPrepareHuggingFaceDatasetJob.TV6Nwm6dFReF/output/data_dir', data_files=None, description=None)
      self.config.name = <local> 'default', len = 7
KeyError: 'default'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.1.0
- Platform: Linux-5.4.0-107-generic-x86_64-with-glibc2.31
- Python version: 3.9.9
- PyArrow version: 6.0.1
- Pandas version: 1.4.2
