Support webdataset format
COLLABORATOR
I like the idea of having `webdataset` as an optional dependency to ensure our loader generates web datasets the same way as the main project.
2022-12-07T16:24:12Z
https://github.com/huggingface/datasets/issues/5337#issuecomment-1341231710
https://api.github.com/repos/huggingface/datasets/issues/5337
IC_kwDODunzps5P8ZJe
https://api.github.com/repos/huggingface/datasets/issues/comments/1341231710/reactions
2022-12-12T14:23:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1341231710
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Webdataset is the one of the most popular dataset formats for large scale computer vision tasks. Upvote for this issue. 
2023-01-04T20:35:31Z
https://github.com/huggingface/datasets/issues/5337#issuecomment-1371391923
https://api.github.com/repos/huggingface/datasets/issues/5337
IC_kwDODunzps5Rvcez
https://api.github.com/repos/huggingface/datasets/issues/comments/1371391923/reactions
2023-01-04T20:35:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1371391923
https://avatars.githubusercontent.com/u/18479688?v=4
https://api.github.com/users/xvjiarui/events{/privacy}
https://api.github.com/users/xvjiarui/followers
https://api.github.com/users/xvjiarui/following{/other_user}
https://api.github.com/users/xvjiarui/gists{/gist_id}
https://github.com/xvjiarui
xvjiarui
MDQ6VXNlcjE4NDc5Njg4
https://api.github.com/users/xvjiarui/orgs
https://api.github.com/users/xvjiarui/received_events
https://api.github.com/users/xvjiarui/repos
https://api.github.com/users/xvjiarui/starred{/owner}{/repo}
https://api.github.com/users/xvjiarui/subscriptions
User
https://api.github.com/users/xvjiarui
public
NONE
Any updates on this?
2023-05-26T05:24:19Z
https://github.com/huggingface/datasets/issues/5337#issuecomment-1563827453
https://api.github.com/repos/huggingface/datasets/issues/5337
IC_kwDODunzps5dNhz9
https://api.github.com/repos/huggingface/datasets/issues/comments/1563827453/reactions
2023-05-26T05:24:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1563827453
https://avatars.githubusercontent.com/u/565317?v=4
https://api.github.com/users/jacobwjs/events{/privacy}
https://api.github.com/users/jacobwjs/followers
https://api.github.com/users/jacobwjs/following{/other_user}
https://api.github.com/users/jacobwjs/gists{/gist_id}
https://github.com/jacobwjs
jacobwjs
MDQ6VXNlcjU2NTMxNw==
https://api.github.com/users/jacobwjs/orgs
https://api.github.com/users/jacobwjs/received_events
https://api.github.com/users/jacobwjs/repos
https://api.github.com/users/jacobwjs/starred{/owner}{/repo}
https://api.github.com/users/jacobwjs/subscriptions
User
https://api.github.com/users/jacobwjs
public
MEMBER
We haven't had the bandwidth to implement it so far, but if someone wants to give it a shot please don't hesitate ^^
2023-05-26T10:34:45Z
https://github.com/huggingface/datasets/issues/5337#issuecomment-1564184054
https://api.github.com/repos/huggingface/datasets/issues/5337
IC_kwDODunzps5dO432
https://api.github.com/repos/huggingface/datasets/issues/comments/1564184054/reactions
2023-05-26T10:34:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1564184054
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Done in #6391 
2024-03-06T14:39:28Z
https://github.com/huggingface/datasets/issues/5337#issuecomment-1981017795
https://api.github.com/repos/huggingface/datasets/issues/5337
IC_kwDODunzps52E-7D
https://api.github.com/repos/huggingface/datasets/issues/comments/1981017795/reactions
2024-03-06T14:39:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1981017795
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Webdataset is an efficient format for iterable datasets. It would be nice to support it in `datasets`, as discussed in https://github.com/rom1504/img2dataset/issues/234.

In particular it would be awesome to be able to load one using `load_dataset` in streaming mode (either from a local directory, or from a dataset on the Hugging Face Hub). Some datasets on the Hub are already in webdataset format.

It terms of implementation, we can have something similar to the Parquet loader.
I also think it's fine to have webdataset as an optional dependency.