Trouble with streaming frgfm/imagenette vision dataset with TAR archive
MEMBER
Hi @frgfm, thanks for reporting.

As the error message says, streaming mode is not supported out of the box when the dataset contains TAR archive files.

To make the dataset streamable, you have to use `dl_manager.iter_archive`.

There are several examples in other datasets, e.g. food101: https://huggingface.co/datasets/food101/blob/main/food101.py

And yes, as the link you pointed out, for the streaming to be possible, the metadata file must be loaded before all of the images:
- either this is the case when iterating the archive (and you get the metadata file before the images)
- or you have to extract the metadata file by hand and upload it separately to the Hub
2022-07-18T07:38:47Z
https://github.com/huggingface/datasets/issues/4697#issuecomment-1186869354
https://api.github.com/repos/huggingface/datasets/issues/4697
IC_kwDODunzps5GvjBq
https://api.github.com/repos/huggingface/datasets/issues/comments/1186869354/reactions
2022-07-18T07:43:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1186869354
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Hi @albertvillanova :wave:

Thanks! Yeah I saw that but since I didn't have any metadata, I wasn't sure whether I should create them myself.

So one last question:
What is the metadata supposed to be for archives? The relative path of all files in it?
_(Sorry I'm a bit confused since it's quite hard to debug using the single error message from the data preview :sweat_smile: )_
2022-07-18T12:28:09Z
https://github.com/huggingface/datasets/issues/4697#issuecomment-1187283096
https://api.github.com/repos/huggingface/datasets/issues/4697
IC_kwDODunzps5GxICY
https://api.github.com/repos/huggingface/datasets/issues/comments/1187283096/reactions
2022-07-18T12:28:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1187283096
https://avatars.githubusercontent.com/u/26927750?v=4
https://api.github.com/users/frgfm/events{/privacy}
https://api.github.com/users/frgfm/followers
https://api.github.com/users/frgfm/following{/other_user}
https://api.github.com/users/frgfm/gists{/gist_id}
https://github.com/frgfm
frgfm
MDQ6VXNlcjI2OTI3NzUw
https://api.github.com/users/frgfm/orgs
https://api.github.com/users/frgfm/received_events
https://api.github.com/users/frgfm/repos
https://api.github.com/users/frgfm/starred{/owner}{/repo}
https://api.github.com/users/frgfm/subscriptions
User
https://api.github.com/users/frgfm
public
MEMBER
Hi @frgfm, streaming a dataset that contains a TAR file requires some tweaks because (contrary to ZIP files), tha TAR archive does not allow random access to any of the contained member files. Instead they have to be accessed sequentially (in the order in which they were put into the TAR file when created) and yielded.

So when iterating over the TAR file content, when an image file is found, we need to yield it (and not keeping it in memory, which will require huge RAM memory for large datasets). But when yielding an image file, we also need to yield with it what we call "metadata": the class label, and other textual information (for example, for audio files, sometimes we also add info such as the speaker ID, their sex, their age,...).

All this information usually is stored in what we call the metadata file: either a JSON or a CSV/TSV file.

But if this is also inside the TAR archive, we need to find this file in the first place when iterating the TAR archive, so that we already have this information when we find an image file and we can yield the image file and its metadata info.

Therefore:
- either the TAR archive contains the metadata file as the first member when iterating it (something we cannot change as it is done at the creation of the TAR file)
- or if not, then we need to have the metadata file elsewhere
  - in these cases, what we do (if the dataset license allows it) is:
    - we download the TAR file locally, we extract the metadata file and we host the metadata on the Hub
    - we modify the dataset loading script so that it first downloads the metadata file (and reads it) and only then starts iterating the content of the TAR archive file

See an example of this process we recently did for "google/fleurs" (their metadata files for "train" were at the end of the TAR archives, after all audio files): https://huggingface.co/datasets/google/fleurs/discussions/4
- we uploaded the metadata file to the Hub
- we adapted the loading script to use it
2022-07-21T13:42:37Z
https://github.com/huggingface/datasets/issues/4697#issuecomment-1191502492
https://api.github.com/repos/huggingface/datasets/issues/4697
IC_kwDODunzps5HBOKc
https://api.github.com/repos/huggingface/datasets/issues/comments/1191502492/reactions
2022-07-21T13:50:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191502492
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Hi @albertvillanova :wave: 

Thanks, since my last message, I went through the repo of https://huggingface.co/datasets/food101/blob/main/food101.py and managed to get it to work in the end :pray: 

Here it is: https://huggingface.co/datasets/frgfm/imagenette

I appreciate you opening an issue to document the process, it might help a few!
2022-07-25T16:15:40Z
https://github.com/huggingface/datasets/issues/4697#issuecomment-1194304494
https://api.github.com/repos/huggingface/datasets/issues/4697
IC_kwDODunzps5HL6Pu
https://api.github.com/repos/huggingface/datasets/issues/comments/1194304494/reactions
2022-07-25T16:15:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1194304494
https://avatars.githubusercontent.com/u/26927750?v=4
https://api.github.com/users/frgfm/events{/privacy}
https://api.github.com/users/frgfm/followers
https://api.github.com/users/frgfm/following{/other_user}
https://api.github.com/users/frgfm/gists{/gist_id}
https://github.com/frgfm
frgfm
MDQ6VXNlcjI2OTI3NzUw
https://api.github.com/users/frgfm/orgs
https://api.github.com/users/frgfm/received_events
https://api.github.com/users/frgfm/repos
https://api.github.com/users/frgfm/starred{/owner}{/repo}
https://api.github.com/users/frgfm/subscriptions
User
https://api.github.com/users/frgfm
public
MEMBER
Great to see that you manage to make your dataset streamable. :rocket: 

I'm closing this issue, as for the docs update there is another issue opened:
- #4711
2022-08-01T15:10:57Z
https://github.com/huggingface/datasets/issues/4697#issuecomment-1201332556
https://api.github.com/repos/huggingface/datasets/issues/4697
IC_kwDODunzps5HmuFM
https://api.github.com/repos/huggingface/datasets/issues/comments/1201332556/reactions
2022-08-01T15:10:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1201332556
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
### Link

https://huggingface.co/datasets/frgfm/imagenette

### Description

Hello there :wave: 

Thanks for the amazing work you've done with HF Datasets! I've just started playing with it, and managed to upload my first dataset. But for the second one, I'm having trouble with the preview since there is some archive extraction involved :sweat_smile: 

Basically, I get a:
```
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://s3.amazonaws.com/fast-ai-imageclas/imagenette2.tgz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

I've tried several things and checked this issue https://github.com/huggingface/datasets/issues/4181 as well, but no luck so far!

Could you point me in the right direction please? :pray: 

### Owner

Yes