Webdataset dataset builder
NONE
_The documentation is not available anymore as the PR was closed or merged._
2023-11-14T14:08:12Z
https://github.com/huggingface/datasets/pull/6391#issuecomment-1810275262
https://api.github.com/repos/huggingface/datasets/issues/6391
IC_kwDODunzps5r5pu-
https://api.github.com/repos/huggingface/datasets/issues/comments/1810275262/reactions
2023-11-28T16:33:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1810275262
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
I added an error message if the first examples don't appear to be in webdataset format
```
"The TAR archives of the dataset should be in Webdataset format, "
"but the files in the archive don't share the same prefix or the same types."
```
2023-11-14T15:50:35Z
https://github.com/huggingface/datasets/pull/6391#issuecomment-1810499106
https://api.github.com/repos/huggingface/datasets/issues/6391
IC_kwDODunzps5r6gYi
https://api.github.com/repos/huggingface/datasets/issues/comments/1810499106/reactions
2023-11-14T15:50:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1810499106
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@mariosasko could you review this ? I think it's fine to have webdataset as an optional dependency for now, then depending on usage and user feedbacks see if it makes sense to have our own implementation or not
2023-11-24T15:32:48Z
https://github.com/huggingface/datasets/pull/6391#issuecomment-1825828019
https://api.github.com/repos/huggingface/datasets/issues/6391
IC_kwDODunzps5s0-yz
https://api.github.com/repos/huggingface/datasets/issues/comments/1825828019/reactions
2023-11-24T15:32:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1825828019
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I just removed the dependency on `webdataset` @mariosasko :)
2023-11-27T18:36:14Z
https://github.com/huggingface/datasets/pull/6391#issuecomment-1828403666
https://api.github.com/repos/huggingface/datasets/issues/6391
IC_kwDODunzps5s-znS
https://api.github.com/repos/huggingface/datasets/issues/comments/1828403666/reactions
2023-11-27T18:36:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1828403666
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
took your comments into account, lmk if you see anything else
2023-11-28T16:03:30Z
https://github.com/huggingface/datasets/pull/6391#issuecomment-1830158303
https://api.github.com/repos/huggingface/datasets/issues/6391
IC_kwDODunzps5tFf_f
https://api.github.com/repos/huggingface/datasets/issues/comments/1830158303/reactions
2023-11-28T16:03:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1830158303
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Allow `load_dataset` to support the Webdataset format.

It allows users to download/stream data from local files or from the Hugging Face Hub.

Moreover it will enable the Dataset Viewer for Webdataset datasets on HF.

## Implementation details

- I added a new Webdataset builder
- dataset with TAR files are now read using the Webdataset builder
- Basic decoding from `webdataset` is used by default, except unsafe ones like pickle
- HF authentication support is done with `xopen`

## TODOS

- [x] tests
- [x] docs