add Speech commands dataset
CONTRIBUTOR
@anton-l ping
2021-11-29T17:40:34Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-981863015
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps46hgpn
https://api.github.com/repos/huggingface/datasets/issues/comments/981863015/reactions
2021-11-29T17:40:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/981863015
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
@lhoestq 
Hi Quentin! Thank you for your feedback and suggestions! 🤗

Yes, that was actually what I wanted to do next - I mean the steaming stuff :)
Also, I need to make some changes to the readme (to account for the updated features set).

Hopefully, I will be done by tomorrow afternoon if that's ok. 

2021-12-01T14:34:07Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-983701124
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps46ohaE
https://api.github.com/repos/huggingface/datasets/issues/comments/983701124/reactions
2021-12-01T14:34:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/983701124
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
@lhoestq  Hi Quentin!

I've implemented (hopefully, correctly) the streaming compatibility but the problem with the current approach is that we first need to iterate over the full archive anyway to get the list of filenames for train and validation sets (see [this](https://github.com/huggingface/datasets/pull/3335/files#diff-aeea540d136025e30a842856779e9c6485a5dc6fc9eb7fd6d3be2acd2f49b8e3R186), the same approach is implemented in TFDS version). Only after that, we can generate examples, so we cannot stream the dataset before the first iteration ends and it takes some time. It's probably not the most effective way. 

If the streaming mode is turned off, this approach (with two iterations) is actually slower than the previous implementation (with archive extraction). 

My suggestion is to host separate archives for each split prepared in advance. That way there would be no need for iterating over the common archive to collect train and validation filenames.  @anton-l  suggested to make AWS mirrors for them. I've prepared these archives, for now you can take a look at them [here](https://drive.google.com/drive/folders/1oMrZHzPgHAKprKJuvih91CM8KMSzh_pL?usp=sharing). I simplified their structure a bit so if we switch to using them, the code then should be changed (and simplified) a bit too.

2021-12-04T10:19:35Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-986004274
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps46xTsy
https://api.github.com/repos/huggingface/datasets/issues/comments/986004274/reactions
2021-12-04T10:58:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/986004274
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
Hi ! Thanks for the changes :)

> My suggestion is to host separate archives for each split prepared in advance. That way there would be no need for iterating over the common archive to collect train and validation filenames. @anton-l suggested to make AWS mirrors for them. I've prepared these archives, for now you can take a look at them here. I simplified their structure a bit so if we switch to using them, the code then should be changed (and simplified) a bit too.

I agree, I just uploaded them on AWS

https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.01/v0.01_test.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.01/v0.01_train.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.01/v0.01_validation.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.02/v0.02_test.tar.gz
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.02/v0.02_validation.tar.gz

Note that in the future we can move those files to actual repositories on the Hugging Face Hub, since we are migrating the datasets from this repository to the Hugging Face Hub (as mirrors), to make them more accessible to the community.
2021-12-06T16:54:12Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-986961482
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps4609ZK
https://api.github.com/repos/huggingface/datasets/issues/comments/986961482/reactions
2021-12-06T16:54:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/986961482
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq Thank you! Gonna look at this tomorrow :)
2021-12-06T21:02:15Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-987206857
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps4615TJ
https://api.github.com/repos/huggingface/datasets/issues/comments/987206857/reactions
2021-12-06T21:02:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/987206857
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
@lhoestq I've modified the code to fit new data format, now it works for v0.01 but doesn't work for v0.02 as the training archive is missing. Could you please create a mirror for that one too? You can find it [here](https://drive.google.com/file/d/1mPjnVMYb-VhPprGlOX8v9TBT1GT-rtcp/view?usp=sharing)

And when it's done I'll need to regenerate all the meta / dummy stuff, and this version will be ready for a review :)
2021-12-07T14:17:51Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-987967446
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps464y_W
https://api.github.com/repos/huggingface/datasets/issues/comments/987967446/reactions
2021-12-07T14:17:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/987967446
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
Here you go :)
https://s3.amazonaws.com/datasets.huggingface.co/SpeechCommands/v0.02/v0.02_train.tar.gz
2021-12-09T11:15:59Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-989757634
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps46_oDC
https://api.github.com/repos/huggingface/datasets/issues/comments/989757634/reactions
2021-12-09T11:15:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/989757634
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
FYI I juste merged a fix for the Windows CI error on `master`, feel free to merge `master` again into your branch
2021-12-09T17:38:00Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-990070976
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps47A0jA
https://api.github.com/repos/huggingface/datasets/issues/comments/990070976/reactions
2021-12-09T17:38:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/990070976
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
All green ! I had to fix some minor stuff in the CI but it's good now

Next step is to mark it as ready for review, and I think it's all good so we can merge 🚀 
2021-12-09T18:07:12Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-990096105
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps47A6rp
https://api.github.com/repos/huggingface/datasets/issues/comments/990096105/reactions
2021-12-09T18:07:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/990096105
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq 🤗
2021-12-09T18:09:37Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-990097937
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps47A7IR
https://api.github.com/repos/huggingface/datasets/issues/comments/990097937/reactions
2021-12-09T18:09:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/990097937
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
:tada: 
2021-12-10T10:31:09Z
https://github.com/huggingface/datasets/pull/3335#issuecomment-990825440
https://api.github.com/repos/huggingface/datasets/issues/3335
IC_kwDODunzps47Dsvg
https://api.github.com/repos/huggingface/datasets/issues/comments/990825440/reactions
2021-12-10T10:31:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/990825440
https://avatars.githubusercontent.com/u/30755778?v=4
https://api.github.com/users/LysandreJik/events{/privacy}
https://api.github.com/users/LysandreJik/followers
https://api.github.com/users/LysandreJik/following{/other_user}
https://api.github.com/users/LysandreJik/gists{/gist_id}
https://github.com/LysandreJik
LysandreJik
MDQ6VXNlcjMwNzU1Nzc4
https://api.github.com/users/LysandreJik/orgs
https://api.github.com/users/LysandreJik/received_events
https://api.github.com/users/LysandreJik/repos
https://api.github.com/users/LysandreJik/starred{/owner}{/repo}
https://api.github.com/users/LysandreJik/subscriptions
User
https://api.github.com/users/LysandreJik
public
closes  #3283