[WIP] Return local paths to Common Voice
CONTRIBUTOR
Cool thanks for giving it a try @anton-l ! 

Would be very much in favor of having "real" paths to the audio files again for non-streaming use cases. At the same time it would be nice to make the audio data loading script as understandable as possible so that the community can easily add audio datasets in the future by looking at this one as an example. Think if it's clear for a contributor how to add an audio datasets script that works for the standard non-streaming case while it is easy to extend it afterwards to a streaming dataset script, then this would be perfect
2022-02-01T22:05:42Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1027332860
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49O9r8
https://api.github.com/repos/huggingface/datasets/issues/comments/1027332860/reactions
2022-02-01T22:05:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1027332860
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
@anton-l @patrickvonplaten @lhoestq Is it possible somehow to provide this logic inside the library instead of a loading script so that we don't need to completely rewrite all the scripts for audio datasets and users don't have to care about two different loading approaches in the same script? 🤔   
2022-02-07T13:46:27Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1031486231
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49ezsX
https://api.github.com/repos/huggingface/datasets/issues/comments/1031486231/reactions
2022-02-07T13:46:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1031486231
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
> @anton-l @patrickvonplaten @lhoestq Is it possible somehow to provide this logic inside the library instead of a loading script so that we don't need to completely rewrite all the scripts for audio datasets and users don't have to care about two different loading approaches in the same script? thinking

Not sure @lhoestq - what do you think? 

Now that we've corrected the previous resampling bug, think this one here is of high importance. @lhoestq - what do you think how we should proceed here? 
2022-02-07T14:39:26Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1031539235
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49fAoj
https://api.github.com/repos/huggingface/datasets/issues/comments/1031539235/reactions
2022-02-07T14:39:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1031539235
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
> @anton-l @patrickvonplaten @lhoestq Is it possible somehow to provide this logic inside the library instead of a loading script so that we don't need to completely rewrite all the scripts for audio datasets and users don't have to care about two different loading approaches in the same script? 🤔

Yes let's do this :)

Maybe we can change the behavior of `DownloadManager.iter_archive` back to extracting the TAR archive locally, and return an iterable of (local path, file obj). And the `StreamingDownloadManager.iter_archive` can return an iterable of (relative path inside the archive, file obj) ?

In this case, a dataset would need to have something like this:
```python
for path, f in files:
    yield id_, {"audio": {"path": path, "bytes": f.read() if not is_local_file(path) else None}}
```

Alternatively, we can allow this if we consider that `Audio.encode_example` sets the "bytes" field to `None` automatically if `path` is a local path:
```python
for path, f in files:
    yield id_, {"audio": {"path": path, "bytes": f.read()}}
```
Note that in this case the file is read for nothing though (maybe it's not a big deal ?)

Let me know if it sounds good to you and what you'd prefer !
2022-02-07T20:02:49Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1031866858
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49gQnq
https://api.github.com/repos/huggingface/datasets/issues/comments/1031866858/reactions
2022-02-08T15:29:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1031866858
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq I'm very much in favor of your first aproach! With the full paths returned I think we won't even need to mess with `os.path.join` vs `"/".join()"` and other local/streaming differences 👍 
2022-02-07T22:43:16Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1032010362
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49gzp6
https://api.github.com/repos/huggingface/datasets/issues/comments/1032010362/reactions
2022-02-07T22:52:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1032010362
https://avatars.githubusercontent.com/u/26864830?v=4
https://api.github.com/users/anton-l/events{/privacy}
https://api.github.com/users/anton-l/followers
https://api.github.com/users/anton-l/following{/other_user}
https://api.github.com/users/anton-l/gists{/gist_id}
https://github.com/anton-l
anton-l
MDQ6VXNlcjI2ODY0ODMw
https://api.github.com/users/anton-l/orgs
https://api.github.com/users/anton-l/received_events
https://api.github.com/users/anton-l/repos
https://api.github.com/users/anton-l/starred{/owner}{/repo}
https://api.github.com/users/anton-l/subscriptions
User
https://api.github.com/users/anton-l
public
COLLABORATOR
@lhoestq I also like the idea and favor your first approach to avoid an unnecessary read and make yielding faster.
2022-02-08T15:11:55Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1032715982
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49jf7O
https://api.github.com/repos/huggingface/datasets/issues/comments/1032715982/reactions
2022-02-08T15:11:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1032715982
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Looks cool - thanks for working on this. I just feel strongly about `path` being an absolute `path` that exist and can be inspected in the non-streaming case :-) For streaming=True IMO it's absolutely fine if we only have access to the bytes
2022-02-08T15:42:22Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1032749906
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49joNS
https://api.github.com/repos/huggingface/datasets/issues/comments/1032749906/reactions
2022-02-08T15:42:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1032749906
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
Hi ! I started implementing this but I noticed that returning an absolute path is breaking for many datasets that do things like
```python
for path, f in files:
    if path.startswith(data_dir):
        ...
```
so I think I will have to add a parameter to `iter_archive` like `extract_locally=True` to avoid the breaking change, does that sound good to you ?

This makes me also think that in streaming mode it could also return a local path too, if we think that writing and deleting temporary files on-the-fly while iterating over the streaming dataset is ok.
2022-02-08T22:58:44Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1033145501
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49lIyd
https://api.github.com/repos/huggingface/datasets/issues/comments/1033145501/reactions
2022-02-08T22:58:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1033145501
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq I think it is a good idea to rollback to extracting the archives locally in non-streaming mode, as far as (as you mentioned) we do not store the bytes in the Arrow file for those cases to avoid "doubling" the disk space usage.

On the other hand, I don't like:
- neither the possibility to avoid extracting locally in non-streaming: the behavior should be consistent; thus we always extract in non-streaming
  - which could be the criterium to decide whether an archive should or should not be extracted? Just because I want to make a condition on path.startswith?
- nor the option to download/delete temporary files in streaming (see discussion here: https://github.com/huggingface/datasets/pull/3689#issuecomment-1032858345)

Unfortunately, in order to fix the datasets that are breaking after the rollback, I would suggest fixing their scripts so that the paths are handled more robustly (considering that they can be absolute or relative).
2022-02-09T07:51:45Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1033450183
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49mTLH
https://api.github.com/repos/huggingface/datasets/issues/comments/1033450183/reactions
2022-02-09T07:57:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1033450183
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I agree with Albert, fixing all of the audio datasets isn't too big of a deal (yet). I can help with those if needed :)
2022-02-09T08:58:11Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1033514305
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49mi1B
https://api.github.com/repos/huggingface/datasets/issues/comments/1033514305/reactions
2022-02-09T08:58:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1033514305
https://avatars.githubusercontent.com/u/26864830?v=4
https://api.github.com/users/anton-l/events{/privacy}
https://api.github.com/users/anton-l/followers
https://api.github.com/users/anton-l/following{/other_user}
https://api.github.com/users/anton-l/gists{/gist_id}
https://github.com/anton-l
anton-l
MDQ6VXNlcjI2ODY0ODMw
https://api.github.com/users/anton-l/orgs
https://api.github.com/users/anton-l/received_events
https://api.github.com/users/anton-l/repos
https://api.github.com/users/anton-l/starred{/owner}{/repo}
https://api.github.com/users/anton-l/subscriptions
User
https://api.github.com/users/anton-l
public
MEMBER
Ok cool ! I'm completely rolling it back then
2022-02-09T23:04:00Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1034290888
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49pgbI
https://api.github.com/repos/huggingface/datasets/issues/comments/1034290888/reactions
2022-02-09T23:04:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1034290888
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Alright I did the rollback and now you can get local paths :)
Feel free to try it out and let me know if it's good for you
2022-02-09T23:53:37Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1034326895
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49ppNv
https://api.github.com/repos/huggingface/datasets/issues/comments/1034326895/reactions
2022-02-09T23:53:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1034326895
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I'll fix the CI tomorrow x)
2022-02-10T01:42:13Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1034397900
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49p6jM
https://api.github.com/repos/huggingface/datasets/issues/comments/1034397900/reactions
2022-02-10T01:42:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1034397900
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Ok according to the CI there around 60+ datasets to fix
2022-02-10T17:54:34Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1035238859
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49tH3L
https://api.github.com/repos/huggingface/datasets/issues/comments/1035238859/reactions
2022-02-10T17:54:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1035238859
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> fixing all of the audio datasets isn't too big of a deal (yet). I can help with those if needed :)

I can help with them too :)

2022-02-11T15:48:11Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1036351235
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49xXcD
https://api.github.com/repos/huggingface/datasets/issues/comments/1036351235/reactions
2022-02-11T15:48:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1036351235
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
Here is the full list to keep track of things:

- [x] air_dialogue
- [x] id_nergrit_corpus
- [ ] id_newspapers_2018
- [x] imdb
- [ ] indic_glue
- [ ] inquisitive_qg
- [x] klue
- [x] lama
- [x] lex_glue
- [ ] lm1b
- [x] amazon_polarity
- [ ] mac_morpho
- [ ] math_dataset
- [ ] md_gender_bias
- [ ] mdd
- [ ] assin
- [ ] atomic
- [ ] babi_qa
- [ ] mlqa
- [ ] mocha
- [ ] blended_skill_talk
- [ ] capes
- [ ] cbt
- [ ] newsgroup
- [ ] cifar10
- [ ] cifar100
- [ ] norec
- [ ] ohsumed
- [ ] code_x_glue_cc_clone_detection_poj104
- [x] openslr
- [ ] orange_sum
- [ ] paws
- [ ] paws-x
- [ ] cppe-5
- [ ] polyglot_ner
- [ ] dbrd
- [ ] empathetic_dialogues
- [ ] eraser_multi_rc
- [ ] flores
- [ ] flue
- [ ] food101
- [ ] py_ast
- [ ] qasc
- [ ] qasper
- [ ] race
- [ ] reuters21578
- [ ] ropes
- [ ] rotten_tomatoes
- [x] vivos
- [ ] wi_locness
- [ ] wiki_movies
- [ ] wikiann
- [ ] wmt20_mlqe_task1
- [ ] wmt20_mlqe_task2
- [ ] wmt20_mlqe_task3
- [ ] scicite
- [ ] xsum
- [ ] scielo
- [ ] scifact
- [ ] setimes
- [ ] social_bias_frames
- [ ] sogou_news
- [x] speech_commands
- [ ] ted_hrlr
- [ ] ted_multi
- [ ] tlc
- [ ] turku_ner_corpus


2022-02-11T21:07:41Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1036623723
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49yZ9r
https://api.github.com/repos/huggingface/datasets/issues/comments/1036623723/reactions
2022-02-15T12:00:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1036623723
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I'll do my best to fix as many as possible tomorrow :)
2022-02-14T19:11:56Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1039454147
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps499M_D
https://api.github.com/repos/huggingface/datasets/issues/comments/1039454147/reactions
2022-02-14T19:11:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1039454147
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
the audio datasets are fixed if I didn't forget anything :)

btw what are we gonna do with the community ones that would be broken after the fix?
2022-02-15T08:15:46Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1039982441
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps49_N9p
https://api.github.com/repos/huggingface/datasets/issues/comments/1039982441/reactions
2022-02-15T08:15:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1039982441
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
Closing in favor of https://github.com/huggingface/datasets/pull/3736
2022-02-22T09:14:05Z
https://github.com/huggingface/datasets/pull/3664#issuecomment-1047579826
https://api.github.com/repos/huggingface/datasets/issues/3664
IC_kwDODunzps4-cMyy
https://api.github.com/repos/huggingface/datasets/issues/comments/1047579826/reactions
2022-02-22T09:14:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1047579826
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Fixes https://github.com/huggingface/datasets/issues/3663

This is a proposed way of returning the old local file-based generator while keeping the new streaming generator intact.

TODO:
- [ ] brainstorm a bit more on https://github.com/huggingface/datasets/issues/3663 to see if we can do better
- [ ] refactor the heck out of this PR to avoid completely copying the logic between the two generators