add support for storage_options for load_dataset API
CONTRIBUTOR
hi @lhoestq,
I saw some errors in my test and found all the failed reasons are `FileNotFoundError`  about `test_load_streaming_private_dataset_with_zipped_data` and `test_load_dataset_private_zipped_images` in `test_load.py `, I run pytest on my own Wins and Ubuntu system all the test in `test_load.py ` are succeed. could you help me to check the test environment of our server?

`2023-06-08T16:50:48.0828281Z FAILED tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data - FileNotFoundError: Couldn't find a dataset script at D:\a\datasets\datasets\__DUMMY_TRANSFORMERS_USER__\repo_zipped_txt_data-16862429577813\repo_zipped_txt_data-16862429577813.py or any data file in the same directory. Couldn't find '__DUMMY_TRANSFORMERS_USER__/repo_zipped_txt_data-16862429577813' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in __DUMMY_TRANSFORMERS_USER__/repo_zipped_txt_data-16862429577813`
`2023-06-08T16:50:48.0830602Z FAILED tests/test_load.py::test_load_dataset_private_zipped_images[False-False] - FileNotFoundError: Couldn't find a dataset script at D:\a\datasets\datasets\__DUMMY_TRANSFORMERS_USER__\repo_zipped_img_data-16862429594168\repo_zipped_img_data-16862429594168.py or any data file in the same directory. Couldn't find '__DUMMY_TRANSFORMERS_USER__/repo_zipped_img_data-16862429594168' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in __DUMMY_TRANSFORMERS_USER__/repo_zipped_img_data-16862429594168`
2023-06-10T20:45:33Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1585812835
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5ehZVj
https://api.github.com/repos/huggingface/datasets/issues/comments/1585812835/reactions
2023-06-10T20:46:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1585812835
https://avatars.githubusercontent.com/u/59083384?v=4
https://api.github.com/users/janineguo/events{/privacy}
https://api.github.com/users/janineguo/followers
https://api.github.com/users/janineguo/following{/other_user}
https://api.github.com/users/janineguo/gists{/gist_id}
https://github.com/janineguo
janineguo
MDQ6VXNlcjU5MDgzMzg0
https://api.github.com/users/janineguo/orgs
https://api.github.com/users/janineguo/received_events
https://api.github.com/users/janineguo/repos
https://api.github.com/users/janineguo/starred{/owner}{/repo}
https://api.github.com/users/janineguo/subscriptions
User
https://api.github.com/users/janineguo
public
MEMBER
I just re-ran the CI, hopefully it's fixed
2023-06-13T11:07:47Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1589081453
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5et3Vt
https://api.github.com/repos/huggingface/datasets/issues/comments/1589081453/reactions
2023-06-13T11:07:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1589081453
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
_The documentation is not available anymore as the PR was closed or merged._
2023-06-13T11:13:52Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1589090107
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5et5c7
https://api.github.com/repos/huggingface/datasets/issues/comments/1589090107/reactions
2023-07-10T09:52:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1589090107
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
> I just re-ran the CI, hopefully it's fixed

I just checked, still has the same error, maybe need someone to fix it
2023-06-13T12:55:07Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1589257967
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5euibv
https://api.github.com/repos/huggingface/datasets/issues/comments/1589257967/reactions
2023-06-13T12:55:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1589257967
https://avatars.githubusercontent.com/u/59083384?v=4
https://api.github.com/users/janineguo/events{/privacy}
https://api.github.com/users/janineguo/followers
https://api.github.com/users/janineguo/following{/other_user}
https://api.github.com/users/janineguo/gists{/gist_id}
https://github.com/janineguo
janineguo
MDQ6VXNlcjU5MDgzMzg0
https://api.github.com/users/janineguo/orgs
https://api.github.com/users/janineguo/received_events
https://api.github.com/users/janineguo/repos
https://api.github.com/users/janineguo/starred{/owner}{/repo}
https://api.github.com/users/janineguo/subscriptions
User
https://api.github.com/users/janineguo
public
MEMBER
I think the issue comes from this PR somehow, since the CI fail is related to loading private repositories and this PR touches authentication related code. Let me check what's the issue, and I'll also review your PR later (sorry I don't have a ton of bandwidth atm)
2023-06-13T15:38:23Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1589562037
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5evsq1
https://api.github.com/repos/huggingface/datasets/issues/comments/1589562037/reactions
2023-06-13T16:11:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1589562037
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5919). All of your documentation changes will be reflected on that endpoint.
2023-07-10T09:58:06Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1628625808
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5hEtuQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1628625808/reactions
2023-07-10T09:58:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1628625808
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
@lhoestq Hi sorry to bother you, the CI check_code_quality failed and it said `would reformat /home/runner/work/datasets/datasets/src/datasets/download/streaming_download_manager.py` but I cant see any changes when I run `python3 -m black --check tests src benchmarks metrics` and `python3 -m ruff tests src benchmarks metrics` on my own computer, is there any version requirements on the tools? I didn't specific the version.
2023-07-11T08:29:47Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1630384751
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5hLbJv
https://api.github.com/repos/huggingface/datasets/issues/comments/1630384751/reactions
2023-07-11T08:30:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1630384751
https://avatars.githubusercontent.com/u/59083384?v=4
https://api.github.com/users/janineguo/events{/privacy}
https://api.github.com/users/janineguo/followers
https://api.github.com/users/janineguo/following{/other_user}
https://api.github.com/users/janineguo/gists{/gist_id}
https://github.com/janineguo
janineguo
MDQ6VXNlcjU5MDgzMzg0
https://api.github.com/users/janineguo/orgs
https://api.github.com/users/janineguo/received_events
https://api.github.com/users/janineguo/repos
https://api.github.com/users/janineguo/starred{/owner}{/repo}
https://api.github.com/users/janineguo/subscriptions
User
https://api.github.com/users/janineguo
public
MEMBER
I just ran `make style` and pushed the changes.
You can install the right versions of black and ruff using `pip install -e .[quality]` ;)
2023-07-11T13:28:20Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1630835614
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5hNJOe
https://api.github.com/repos/huggingface/datasets/issues/comments/1630835614/reactions
2023-07-11T13:28:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1630835614
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I am working on this issue right now https://github.com/huggingface/datasets/issues/6017 which is strongly connected to your PR, and I might end up cherry-picking some of your commits (keeping attribution of course !). Would you be ok with that ?
2023-07-12T10:45:30Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1632274377
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5hSofJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1632274377/reactions
2023-07-12T10:45:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1632274377
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
it's totally ok for me, I just wish the S3 File system could support streaming too.

2023-07-13T05:43:17Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1633586236
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5hXow8
https://api.github.com/repos/huggingface/datasets/issues/comments/1633586236/reactions
2023-07-13T05:43:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1633586236
https://avatars.githubusercontent.com/u/59083384?v=4
https://api.github.com/users/janineguo/events{/privacy}
https://api.github.com/users/janineguo/followers
https://api.github.com/users/janineguo/following{/other_user}
https://api.github.com/users/janineguo/gists{/gist_id}
https://github.com/janineguo
janineguo
MDQ6VXNlcjU5MDgzMzg0
https://api.github.com/users/janineguo/orgs
https://api.github.com/users/janineguo/received_events
https://api.github.com/users/janineguo/repos
https://api.github.com/users/janineguo/starred{/owner}{/repo}
https://api.github.com/users/janineguo/subscriptions
User
https://api.github.com/users/janineguo
public
CONTRIBUTOR

I already adjust the code and test on my local Mac, you can check it now, and you can make any changes to it.
2023-07-13T05:52:32Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1633593763
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5hXqmj
https://api.github.com/repos/huggingface/datasets/issues/comments/1633593763/reactions
2023-07-13T05:52:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1633593763
https://avatars.githubusercontent.com/u/59083384?v=4
https://api.github.com/users/janineguo/events{/privacy}
https://api.github.com/users/janineguo/followers
https://api.github.com/users/janineguo/following{/other_user}
https://api.github.com/users/janineguo/gists{/gist_id}
https://github.com/janineguo
janineguo
MDQ6VXNlcjU5MDgzMzg0
https://api.github.com/users/janineguo/orgs
https://api.github.com/users/janineguo/received_events
https://api.github.com/users/janineguo/repos
https://api.github.com/users/janineguo/starred{/owner}{/repo}
https://api.github.com/users/janineguo/subscriptions
User
https://api.github.com/users/janineguo
public
MEMBER
Closing this PR in favor of https://github.com/huggingface/datasets/pull/6028 which includes your contribution :)
2023-07-17T17:02:00Z
https://github.com/huggingface/datasets/pull/5919#issuecomment-1638528845
https://api.github.com/repos/huggingface/datasets/issues/5919
IC_kwDODunzps5hqfdN
https://api.github.com/repos/huggingface/datasets/issues/comments/1638528845/reactions
2023-07-17T17:02:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1638528845
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
to solve the issue in #5880 

1.  add s3 support in the link check step, previous we only check `http` and `https`,

2. change the parameter of `use_auth_token` to `download_config` to support both `storage_options` and `use_auth_token` parameter when trying to handle(list, open, read, etc,.) the remote files.

3.  integrate the check part's duplicate code to make adding or deleting other sources easier.