Support streaming FLEURS dataset
MEMBER
Yes, you just have to use `dl_manager.iter_archive` instead of `dl_manager.download_and_extract`.

That's because `download_and_extract` doesn't support TAR archives in streaming mode.
2022-04-22T12:40:08Z
https://github.com/huggingface/datasets/issues/4181#issuecomment-1106474205
https://api.github.com/repos/huggingface/datasets/issues/4181
IC_kwDODunzps5B83Td
https://api.github.com/repos/huggingface/datasets/issues/comments/1106474205/reactions
2022-04-22T12:40:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1106474205
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Tried to make it streamable, but I don't think it's really possible. @lhoestq @polinaeterna maybe you guys can check: 
https://huggingface.co/datasets/google/fleurs/commit/dcf80160cd77977490a8d32b370c027107f2407b 

real quick. 

I think the problem is that we cannot ensure that the metadata file is found before the audio. Or is this possible somehow @lhoestq ? 
2022-04-25T13:56:23Z
https://github.com/huggingface/datasets/issues/4181#issuecomment-1108612208
https://api.github.com/repos/huggingface/datasets/issues/4181
IC_kwDODunzps5CFBRw
https://api.github.com/repos/huggingface/datasets/issues/comments/1108612208/reactions
2022-04-25T13:56:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1108612208
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
@patrickvonplaten I think the metadata file should be found first because the audio files are contained in a folder next to the metadata files (just as in common voice), so the metadata files should be "on top of the list"   as they are closer to the root in the directories hierarchy 
2022-04-25T16:20:08Z
https://github.com/huggingface/datasets/issues/4181#issuecomment-1108780644
https://api.github.com/repos/huggingface/datasets/issues/4181
IC_kwDODunzps5CFqZk
https://api.github.com/repos/huggingface/datasets/issues/comments/1108780644/reactions
2022-04-25T16:33:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1108780644
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
@patrickvonplaten but apparently it doesn't... I don't really know why.
2022-04-25T16:32:56Z
https://github.com/huggingface/datasets/issues/4181#issuecomment-1108792975
https://api.github.com/repos/huggingface/datasets/issues/4181
IC_kwDODunzps5CFtaP
https://api.github.com/repos/huggingface/datasets/issues/comments/1108792975/reactions
2022-04-25T16:32:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1108792975
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
Yeah! Any ideas what could be the reason here? cc @lhoestq ?
2022-04-25T20:42:27Z
https://github.com/huggingface/datasets/issues/4181#issuecomment-1109021363
https://api.github.com/repos/huggingface/datasets/issues/4181
IC_kwDODunzps5CGlKz
https://api.github.com/repos/huggingface/datasets/issues/comments/1109021363/reactions
2022-04-25T20:42:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1109021363
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
The order of the files is determined when the TAR archive is created, depending on the commands the creator ran.
If the metadata file is not at the beginning of the file, that makes streaming completely inefficient. In this case the TAR archive needs to be recreated in an appropriate order.
2022-04-27T12:06:33Z
https://github.com/huggingface/datasets/issues/4181#issuecomment-1110920559
https://api.github.com/repos/huggingface/datasets/issues/4181
IC_kwDODunzps5CN01v
https://api.github.com/repos/huggingface/datasets/issues/comments/1110920559/reactions
2022-04-27T12:06:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1110920559
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Actually we could maybe just host the metadata file ourselves and then stream the audio data only. Don't think that this would be a problem for the FLEURS authors (I can ask them :-)) 
2022-04-27T21:29:10Z
https://github.com/huggingface/datasets/issues/4181#issuecomment-1111501934
https://api.github.com/repos/huggingface/datasets/issues/4181
IC_kwDODunzps5CQCxu
https://api.github.com/repos/huggingface/datasets/issues/comments/1111501934/reactions
2022-04-27T21:29:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1111501934
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
I made a PR to their repo to support streaming (by uploading the metadata file to the Hub). See:
- https://huggingface.co/datasets/google/fleurs/discussions/4
2022-07-19T14:39:29Z
https://github.com/huggingface/datasets/issues/4181#issuecomment-1189139300
https://api.github.com/repos/huggingface/datasets/issues/4181
IC_kwDODunzps5G4NNk
https://api.github.com/repos/huggingface/datasets/issues/comments/1189139300/reactions
2022-07-19T14:39:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1189139300
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I'm closing this issue as the PR above has been merged.
2022-07-25T11:44:02Z
https://github.com/huggingface/datasets/issues/4181#issuecomment-1193943595
https://api.github.com/repos/huggingface/datasets/issues/4181
IC_kwDODunzps5HKiIr
https://api.github.com/repos/huggingface/datasets/issues/comments/1193943595/reactions
2022-07-25T11:44:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1193943595
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
## Dataset viewer issue for '*name of the dataset*'

https://huggingface.co/datasets/google/fleurs

```
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://storage.googleapis.com/xtreme_translations/FLEURS/af_za.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

Am I the one who added this dataset ? Yes

Can I fix this somehow in the script? @lhoestq @severo 
