Support cloud storage in load_dataset
NONE
Or for example an archive on GitHub releases!  Before I added support for JXL (locally only, PR still pending) I was considering hosting my files on GitHub instead...
2022-11-25T15:54:18Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1327649414
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5PIlKG
https://api.github.com/repos/huggingface/datasets/issues/comments/1327649414/reactions
2022-11-25T15:54:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1327649414
https://avatars.githubusercontent.com/u/445208?v=4
https://api.github.com/users/alexjc/events{/privacy}
https://api.github.com/users/alexjc/followers
https://api.github.com/users/alexjc/following{/other_user}
https://api.github.com/users/alexjc/gists{/gist_id}
https://github.com/alexjc
alexjc
MDQ6VXNlcjQ0NTIwOA==
https://api.github.com/users/alexjc/orgs
https://api.github.com/users/alexjc/received_events
https://api.github.com/users/alexjc/repos
https://api.github.com/users/alexjc/starred{/owner}{/repo}
https://api.github.com/users/alexjc/subscriptions
User
https://api.github.com/users/alexjc
public
NONE
+1 to this. I would like to use 'audiofolder' with a data_dir that's on S3, for example. I don't want to upload my dataset to the Hub, but I would find all the fingerprinting/caching features useful.
2022-12-09T03:58:11Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1343813555
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5QGPez
https://api.github.com/repos/huggingface/datasets/issues/comments/1343813555/reactions
2022-12-09T03:58:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1343813555
https://avatars.githubusercontent.com/u/933687?v=4
https://api.github.com/users/iceboundflame/events{/privacy}
https://api.github.com/users/iceboundflame/followers
https://api.github.com/users/iceboundflame/following{/other_user}
https://api.github.com/users/iceboundflame/gists{/gist_id}
https://github.com/iceboundflame
iceboundflame
MDQ6VXNlcjkzMzY4Nw==
https://api.github.com/users/iceboundflame/orgs
https://api.github.com/users/iceboundflame/received_events
https://api.github.com/users/iceboundflame/repos
https://api.github.com/users/iceboundflame/starred{/owner}{/repo}
https://api.github.com/users/iceboundflame/subscriptions
User
https://api.github.com/users/iceboundflame
public
CONTRIBUTOR
Adding to the conversation, Dask also uses `fsspec` for this feature.

[Dask: How to connect to remote data](https://docs.dask.org/en/stable/how-to/connect-to-remote-data.html)

Happy to help on this feature :D 
2022-12-12T14:21:07Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1346593507
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5QQ2Lj
https://api.github.com/repos/huggingface/datasets/issues/comments/1346593507/reactions
2022-12-12T15:00:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1346593507
https://avatars.githubusercontent.com/u/8976546?v=4
https://api.github.com/users/Dref360/events{/privacy}
https://api.github.com/users/Dref360/followers
https://api.github.com/users/Dref360/following{/other_user}
https://api.github.com/users/Dref360/gists{/gist_id}
https://github.com/Dref360
Dref360
MDQ6VXNlcjg5NzY1NDY=
https://api.github.com/users/Dref360/orgs
https://api.github.com/users/Dref360/received_events
https://api.github.com/users/Dref360/repos
https://api.github.com/users/Dref360/starred{/owner}{/repo}
https://api.github.com/users/Dref360/subscriptions
User
https://api.github.com/users/Dref360
public
NONE
+1 to this feature request since I think it also tackles my use-case. I am collaborating with a team, working with a loading script which takes some time to generate the dataset artifacts. It would be very handy to use this as a cloud cache to avoid duplicating the effort. 

Currently we could use `builder.download_and_prepare(path_to_cloud_storage, storage_options, ...)`  to cache the artifacts to cloud storage, but then `builder.as_dataset()` yields `NotImplementedError: Loading a dataset cached in SomeCloudFileSystem is not supported`
2023-02-08T11:20:47Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1422436413
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5UyKg9
https://api.github.com/repos/huggingface/datasets/issues/comments/1422436413/reactions
2023-02-08T11:20:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1422436413
https://avatars.githubusercontent.com/u/44843469?v=4
https://api.github.com/users/eballesteros/events{/privacy}
https://api.github.com/users/eballesteros/followers
https://api.github.com/users/eballesteros/following{/other_user}
https://api.github.com/users/eballesteros/gists{/gist_id}
https://github.com/eballesteros
eballesteros
MDQ6VXNlcjQ0ODQzNDY5
https://api.github.com/users/eballesteros/orgs
https://api.github.com/users/eballesteros/received_events
https://api.github.com/users/eballesteros/repos
https://api.github.com/users/eballesteros/starred{/owner}{/repo}
https://api.github.com/users/eballesteros/subscriptions
User
https://api.github.com/users/eballesteros
public
MEMBER
Makes sense ! If you want to load locally a dataset that you download_and_prepared on a cloud storage, you would use `load_dataset(path_to_cloud_storage)` indeed. It would download the data from the cloud storage, cache them locally, and return a `Dataset`.
2023-02-08T13:03:37Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1422563626
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5Uypkq
https://api.github.com/repos/huggingface/datasets/issues/comments/1422563626/reactions
2023-02-08T13:03:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1422563626
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
It seems currently the `cached_path` function handles all URLs by `get_from_cache` that only supports `ftp` and `http(s)` here:
https://github.com/huggingface/datasets/blob/b5672a956d5de864e6f5550e493527d962d6ae55/src/datasets/utils/file_utils.py#L181

I guess one can add another condition that handles `s3://` or `gs://` URLs via `fsspec` here.
2023-02-16T08:11:48Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1432683648
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5VZQSA
https://api.github.com/repos/huggingface/datasets/issues/comments/1432683648/reactions
2023-02-16T08:11:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1432683648
https://avatars.githubusercontent.com/u/1190780?v=4
https://api.github.com/users/kyamagu/events{/privacy}
https://api.github.com/users/kyamagu/followers
https://api.github.com/users/kyamagu/following{/other_user}
https://api.github.com/users/kyamagu/gists{/gist_id}
https://github.com/kyamagu
kyamagu
MDQ6VXNlcjExOTA3ODA=
https://api.github.com/users/kyamagu/orgs
https://api.github.com/users/kyamagu/received_events
https://api.github.com/users/kyamagu/repos
https://api.github.com/users/kyamagu/starred{/owner}{/repo}
https://api.github.com/users/kyamagu/subscriptions
User
https://api.github.com/users/kyamagu
public
CONTRIBUTOR
I could use this functionality, so I put together a PR using @kyamagu's suggestion to use `fsspec` in `datasets.utils.file_utils`

https://github.com/huggingface/datasets/pull/5580
2023-02-27T04:10:27Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1445674952
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5WKz_I
https://api.github.com/repos/huggingface/datasets/issues/comments/1445674952/reactions
2023-02-27T04:10:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1445674952
https://avatars.githubusercontent.com/u/2512762?v=4
https://api.github.com/users/dwyatte/events{/privacy}
https://api.github.com/users/dwyatte/followers
https://api.github.com/users/dwyatte/following{/other_user}
https://api.github.com/users/dwyatte/gists{/gist_id}
https://github.com/dwyatte
dwyatte
MDQ6VXNlcjI1MTI3NjI=
https://api.github.com/users/dwyatte/orgs
https://api.github.com/users/dwyatte/received_events
https://api.github.com/users/dwyatte/repos
https://api.github.com/users/dwyatte/starred{/owner}{/repo}
https://api.github.com/users/dwyatte/subscriptions
User
https://api.github.com/users/dwyatte
public
MEMBER
Thanks @dwyatte for adding support for fsspec urls

Let me just reopen this since the original issue is not resolved
2023-03-11T00:56:37Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1464732202
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5XTgoq
https://api.github.com/repos/huggingface/datasets/issues/comments/1464732202/reactions
2023-04-05T09:01:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1464732202
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I'm not yet understanding how to use https://github.com/huggingface/datasets/pull/5580 in order to use `load_dataset(data_files="s3://...")`. Any help/example would be much appreciated :) thanks! 
2023-03-22T12:06:54Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1479460556
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5YLsbM
https://api.github.com/repos/huggingface/datasets/issues/comments/1479460556/reactions
2023-03-22T12:06:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1479460556
https://avatars.githubusercontent.com/u/3419333?v=4
https://api.github.com/users/janmaltel/events{/privacy}
https://api.github.com/users/janmaltel/followers
https://api.github.com/users/janmaltel/following{/other_user}
https://api.github.com/users/janmaltel/gists{/gist_id}
https://github.com/janmaltel
janmaltel
MDQ6VXNlcjM0MTkzMzM=
https://api.github.com/users/janmaltel/orgs
https://api.github.com/users/janmaltel/received_events
https://api.github.com/users/janmaltel/repos
https://api.github.com/users/janmaltel/starred{/owner}{/repo}
https://api.github.com/users/janmaltel/subscriptions
User
https://api.github.com/users/janmaltel
public
MEMBER
It's still not officially supported x) But you can try to update `request_etag` in `file_utils.py` to use `fsspec_head` instead of `http_head`. It is responsible of getting the ETags of the remote files for caching. This change may do the trick for S3 urls
2023-03-22T13:29:15Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1479569883
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5YMHHb
https://api.github.com/repos/huggingface/datasets/issues/comments/1479569883/reactions
2023-03-22T13:29:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1479569883
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thank you for your guys help on this and merging in #5580. I manually pulled the changes to my local datasets package (datasets.utils.file_utils.py) since it only seemed to be this file that was changed in the PR and I'm getting the error: 
InvalidSchema: No connection adapters were found for 's3://bucket/folder/'. I'm calling load_dataset using the S3 URI. When I use the S3 URL I get HTTPError: 403 Client Error. 
Am I not supposed to use the S3 URI? How do I pull in the changes from this merge? I'm running datasets 2.10.1. 
2023-03-24T17:07:36Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1483136648
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5YZt6I
https://api.github.com/repos/huggingface/datasets/issues/comments/1483136648/reactions
2023-03-24T17:07:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1483136648
https://avatars.githubusercontent.com/u/34521882?v=4
https://api.github.com/users/ssabatier/events{/privacy}
https://api.github.com/users/ssabatier/followers
https://api.github.com/users/ssabatier/following{/other_user}
https://api.github.com/users/ssabatier/gists{/gist_id}
https://github.com/ssabatier
ssabatier
MDQ6VXNlcjM0NTIxODgy
https://api.github.com/users/ssabatier/orgs
https://api.github.com/users/ssabatier/received_events
https://api.github.com/users/ssabatier/repos
https://api.github.com/users/ssabatier/starred{/owner}{/repo}
https://api.github.com/users/ssabatier/subscriptions
User
https://api.github.com/users/ssabatier
public
CONTRIBUTOR
The current implementation depends on gcsfs/s3fs being able to authenticate through some other means e.g., environmental variables. For AWS, it looks like you can set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`

Note that while testing this just now, I did note a discrepancy between gcsfs and  s3fs that we might want to address where gcsfs passes the timeout from `storage_options` [here](https://github.com/huggingface/datasets/blob/3e6269979fc80ae8939294d26298897f0db5b84d/src/datasets/utils/file_utils.py#L333) down into the `aiohttp.ClientSession.request`, but s3fs does not handle this (tries to pass to the `aiobotocore.session.AioSession` constructor raising `TypeError: __init__() got an unexpected keyword argument 'requests_timeout'`).

It seems like some work trying to unify kwargs across different fsspec implementations, so if the plan is to pass down `storage_options`, I wonder if we should just let users control the timeout (and other kwargs) using that and if not specified, use the default?
2023-03-26T16:56:11Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1484156837
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5Ydm-l
https://api.github.com/repos/huggingface/datasets/issues/comments/1484156837/reactions
2023-03-27T00:57:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1484156837
https://avatars.githubusercontent.com/u/2512762?v=4
https://api.github.com/users/dwyatte/events{/privacy}
https://api.github.com/users/dwyatte/followers
https://api.github.com/users/dwyatte/following{/other_user}
https://api.github.com/users/dwyatte/gists{/gist_id}
https://github.com/dwyatte
dwyatte
MDQ6VXNlcjI1MTI3NjI=
https://api.github.com/users/dwyatte/orgs
https://api.github.com/users/dwyatte/received_events
https://api.github.com/users/dwyatte/repos
https://api.github.com/users/dwyatte/starred{/owner}{/repo}
https://api.github.com/users/dwyatte/subscriptions
User
https://api.github.com/users/dwyatte
public
CONTRIBUTOR
> Note that while testing this just now, I did note a discrepancy between gcsfs and s3fs that we might want to address where gcsfs passes the timeout from storage_options [here](https://github.com/huggingface/datasets/blob/3e6269979fc80ae8939294d26298897f0db5b84d/src/datasets/utils/file_utils.py#L333) down into the aiohttp.ClientSession.request, but s3fs does not handle this (tries to pass to the aiobotocore.session.AioSession constructor raising TypeError: __init__() got an unexpected keyword argument 'requests_timeout').

> It seems like some work trying to unify kwargs across different fsspec implementations, so if the plan is to pass down storage_options, I wonder if we should just let users control the timeout (and other kwargs) and if not specified, use the default?

@lhoestq here's a small PR for this: https://github.com/huggingface/datasets/pull/5673


2023-03-26T20:09:44Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1484209288
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5YdzyI
https://api.github.com/repos/huggingface/datasets/issues/comments/1484209288/reactions
2023-03-26T20:17:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1484209288
https://avatars.githubusercontent.com/u/2512762?v=4
https://api.github.com/users/dwyatte/events{/privacy}
https://api.github.com/users/dwyatte/followers
https://api.github.com/users/dwyatte/following{/other_user}
https://api.github.com/users/dwyatte/gists{/gist_id}
https://github.com/dwyatte
dwyatte
MDQ6VXNlcjI1MTI3NjI=
https://api.github.com/users/dwyatte/orgs
https://api.github.com/users/dwyatte/received_events
https://api.github.com/users/dwyatte/repos
https://api.github.com/users/dwyatte/starred{/owner}{/repo}
https://api.github.com/users/dwyatte/subscriptions
User
https://api.github.com/users/dwyatte
public
NONE
@lhoestq sorry for being a little dense here but I am very keen to use fsspec / adlfs for for a larger image dataset I have for object detection. I have to keep it on Azure storage and would also like to avoid a full download or zipping (so use `load_dataset(..., streaming=True)`. So this development is godsend :) only... I am unable to make it work.

Would you expect the setup to work for:
- azure blob storage
- image files (not the standard formats `json`, `parquet`....

? I appreciate that you mostly focus on s3 but it seems that, similar to the [remaining cloud storage functionality](https://huggingface.co/docs/datasets/filesystems#cloud-storage), it should also work for Azure blob storage.


I would imagine that something like (Streaming true or false):
```python
d = load_dataset("new_dataset.py", storage_options=storage_options, split="train")
```
would work with 
```python
# new_dataset.py
....
_URL="abfs://container/image_folder``` 

archive_path = dl_manager.download(_URL)
split_metadata_paths = dl_manager.download(_METADATA_URLS)
return [
    datasets.SplitGenerator(
        name=datasets.Split.TRAIN,
        gen_kwargs={
            "annotation_file_path": split_metadata_paths["train"],
            "files": dl_manager.iter_files(archive_path)
},
      ),
...
```
but I get 
```
Traceback (most recent call last):
...        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/builder.py", line 1649, in _download_and_prepare
    super()._download_and_prepare(
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/builder.py", line 963, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.cache/huggingface/modules/datasets_modules/datasets/new_dataset/dd26a081eab90074f41fa2c821b458424fde393cc73d3d8241aca956d1fb3aa0/new_dataset_script.py", line 56, in _split_generators
    archive_path = dl_manager.download(_URL)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/download/download_manager.py", line 427, in download
    downloaded_path_or_paths = map_nested(
                               ^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/utils/py_utils.py", line 435, in map_nested
    return function(data_struct)
           ^^^^^^^^^^^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/download/download_manager.py", line 453, in _download
    return cached_path(url_or_filename, download_config=download_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 206, in cached_path
    raise ValueError(f"unable to parse {url_or_filename} as a URL or as a local path")
ValueError: unable to parse abfs://container/image_folder as a URL or as a local path

```

2023-08-07T11:51:33Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1667712680
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5jZ0ao
https://api.github.com/repos/huggingface/datasets/issues/comments/1667712680/reactions
2023-08-07T11:53:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1667712680
https://avatars.githubusercontent.com/u/126508264?v=4
https://api.github.com/users/mayorblock/events{/privacy}
https://api.github.com/users/mayorblock/followers
https://api.github.com/users/mayorblock/following{/other_user}
https://api.github.com/users/mayorblock/gists{/gist_id}
https://github.com/mayorblock
mayorblock
U_kgDOB4pc6A
https://api.github.com/users/mayorblock/orgs
https://api.github.com/users/mayorblock/received_events
https://api.github.com/users/mayorblock/repos
https://api.github.com/users/mayorblock/starred{/owner}{/repo}
https://api.github.com/users/mayorblock/subscriptions
User
https://api.github.com/users/mayorblock
public
MEMBER
What version of `datasets` are you using ?
2023-08-16T09:57:03Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1680311709
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5kJ4Wd
https://api.github.com/repos/huggingface/datasets/issues/comments/1680311709/reactions
2023-08-16T09:57:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1680311709
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq 
hello, i still have problem with loading json from S3:


storage_options = {
    "key": xxxx,
    "secret": xxx,
    "endpoint_url": xxxx
}
path = 's3://xxx/xxxxxxx.json'
dataset = load_dataset("json", data_files=path, storage_options=storage_options)

and it throws an error:
TypeError: AioSession.__init__() got an unexpected keyword argument 'hf'
and I use the lastest 2.14.4_dev0 version
2023-08-17T05:15:18Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1681626645
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5kO5YV
https://api.github.com/repos/huggingface/datasets/issues/comments/1681626645/reactions
2023-08-17T05:19:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1681626645
https://avatars.githubusercontent.com/u/37239015?v=4
https://api.github.com/users/hjq133/events{/privacy}
https://api.github.com/users/hjq133/followers
https://api.github.com/users/hjq133/following{/other_user}
https://api.github.com/users/hjq133/gists{/gist_id}
https://github.com/hjq133
hjq133
MDQ6VXNlcjM3MjM5MDE1
https://api.github.com/users/hjq133/orgs
https://api.github.com/users/hjq133/received_events
https://api.github.com/users/hjq133/repos
https://api.github.com/users/hjq133/starred{/owner}{/repo}
https://api.github.com/users/hjq133/subscriptions
User
https://api.github.com/users/hjq133
public
NONE
Hi @lhoestq, thanks for getting back to me :) you have been busy over the summer I see... I was on `2.12.0`. I have updated to `2.14.4`. 

Now `d = load_dataset("new_dataset.py", storage_options=storage_options, split="train", streaming=True)` works for Azure blob storage (with a local data loader script) when I explicitly list all blobs  (I am struggling to make `fs.ls(<path>)` work in the script to make the list available to the download manager).

 Any chance that it could work out-of-the-box by supplying just the image folder, not the full list of image filenames? It seems that `dl_manager.download(_URL)` always wants one or more (possibly archived) files. In my situation, where I don't want to archive or download, it would be great to just supply the folder (seems reasonably doable with fsspec).

Let me know if there is anything I can do to help.

Thanks,
2023-08-17T06:30:37Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1681691402
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5kPJMK
https://api.github.com/repos/huggingface/datasets/issues/comments/1681691402/reactions
2023-08-17T06:30:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1681691402
https://avatars.githubusercontent.com/u/126508264?v=4
https://api.github.com/users/mayorblock/events{/privacy}
https://api.github.com/users/mayorblock/followers
https://api.github.com/users/mayorblock/following{/other_user}
https://api.github.com/users/mayorblock/gists{/gist_id}
https://github.com/mayorblock
mayorblock
U_kgDOB4pc6A
https://api.github.com/users/mayorblock/orgs
https://api.github.com/users/mayorblock/received_events
https://api.github.com/users/mayorblock/repos
https://api.github.com/users/mayorblock/starred{/owner}{/repo}
https://api.github.com/users/mayorblock/subscriptions
User
https://api.github.com/users/mayorblock
public
MEMBER
> Any chance that it could work out-of-the-box by supplying just the image folder, not the full list of image filenames? It seems that dl_manager.download(_URL) always wants one or more (possibly archived) files. In my situation, where I don't want to archive or download, it would be great to just supply the folder (seems reasonably doable with fsspec).

@mayorblock  This is not supported right now, you have to use archives or implement a way to get the list by yourself

> TypeError: AioSession.init() got an unexpected keyword argument 'hf'

@hjq133 Can you update `fsspec` and try again ?

```python
pip install -U fsspec
```
2023-08-17T08:23:44Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1681851459
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5kPwRD
https://api.github.com/repos/huggingface/datasets/issues/comments/1681851459/reactions
2023-08-17T08:23:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1681851459
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
thanks for your suggestion，it works now ! 
2023-08-18T13:04:11Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1683891083
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5kXiOL
https://api.github.com/repos/huggingface/datasets/issues/comments/1683891083/reactions
2023-08-18T13:04:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1683891083
https://avatars.githubusercontent.com/u/37239015?v=4
https://api.github.com/users/hjq133/events{/privacy}
https://api.github.com/users/hjq133/followers
https://api.github.com/users/hjq133/following{/other_user}
https://api.github.com/users/hjq133/gists{/gist_id}
https://github.com/hjq133
hjq133
MDQ6VXNlcjM3MjM5MDE1
https://api.github.com/users/hjq133/orgs
https://api.github.com/users/hjq133/received_events
https://api.github.com/users/hjq133/repos
https://api.github.com/users/hjq133/starred{/owner}{/repo}
https://api.github.com/users/hjq133/subscriptions
User
https://api.github.com/users/hjq133
public
NONE
I'm seeing same problem as @hjq133 with following versions:

```
datasets==2.15.0
(venv) ➜  finetuning-llama2 git:(main) ✗ pip freeze | grep s3fs   
s3fs==2023.10.0
(venv) ➜  finetuning-llama2 git:(main) ✗ pip freeze | grep fsspec
fsspec==2023.10.0
```
2023-11-23T14:16:35Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1824510795
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5sv9NL
https://api.github.com/repos/huggingface/datasets/issues/comments/1824510795/reactions
2023-11-23T14:16:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1824510795
https://avatars.githubusercontent.com/u/1917451?v=4
https://api.github.com/users/mariokostelac/events{/privacy}
https://api.github.com/users/mariokostelac/followers
https://api.github.com/users/mariokostelac/following{/other_user}
https://api.github.com/users/mariokostelac/gists{/gist_id}
https://github.com/mariokostelac
mariokostelac
MDQ6VXNlcjE5MTc0NTE=
https://api.github.com/users/mariokostelac/orgs
https://api.github.com/users/mariokostelac/received_events
https://api.github.com/users/mariokostelac/repos
https://api.github.com/users/mariokostelac/starred{/owner}{/repo}
https://api.github.com/users/mariokostelac/subscriptions
User
https://api.github.com/users/mariokostelac
public
NONE
> @lhoestq hello, i still have problem with loading json from S3:
> 
> storage_options = { "key": xxxx, "secret": xxx, "endpoint_url": xxxx } path = 's3://xxx/xxxxxxx.json' dataset = load_dataset("json", data_files=path, storage_options=storage_options)
> 
> and it throws an error: TypeError: AioSession.**init**() got an unexpected keyword argument 'hf' and I use the lastest 2.14.4_dev0 version

I am trying to do the same thing, but the loading is just hanging, without any error. 
@lhoestq is there any documentation how to load from private s3 buckets?

2023-11-26T13:33:12Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1826785910
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5s4op2
https://api.github.com/repos/huggingface/datasets/issues/comments/1826785910/reactions
2023-11-26T13:33:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1826785910
https://avatars.githubusercontent.com/u/14240313?v=4
https://api.github.com/users/aarbelle/events{/privacy}
https://api.github.com/users/aarbelle/followers
https://api.github.com/users/aarbelle/following{/other_user}
https://api.github.com/users/aarbelle/gists{/gist_id}
https://github.com/aarbelle
aarbelle
MDQ6VXNlcjE0MjQwMzEz
https://api.github.com/users/aarbelle/orgs
https://api.github.com/users/aarbelle/received_events
https://api.github.com/users/aarbelle/repos
https://api.github.com/users/aarbelle/starred{/owner}{/repo}
https://api.github.com/users/aarbelle/subscriptions
User
https://api.github.com/users/aarbelle
public
MEMBER
Hi ! S3 support is still experimental. It seems like there is an extra `hf` field passed to the `s3fs` storage_options that causes this error. I just check the source code of `_prepare_single_hop_path_and_storage_options` and I think you can try passing explicitly your own `storage_options={"s3": {...}}`. Also note that it's generally better to load datasets from HF (we run extensive tests and benchmarks for speed and robustness)
2023-11-27T11:13:25Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1827629471
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5s72mf
https://api.github.com/repos/huggingface/datasets/issues/comments/1827629471/reactions
2023-11-27T11:13:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1827629471
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
That worked! Thanks
It seems thought that `data_dir=...` doesn't work on s3, only `data_files`.

2023-11-28T07:26:18Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1829250845
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5tCCcd
https://api.github.com/repos/huggingface/datasets/issues/comments/1829250845/reactions
2023-11-28T07:26:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1829250845
https://avatars.githubusercontent.com/u/14240313?v=4
https://api.github.com/users/aarbelle/events{/privacy}
https://api.github.com/users/aarbelle/followers
https://api.github.com/users/aarbelle/following{/other_user}
https://api.github.com/users/aarbelle/gists{/gist_id}
https://github.com/aarbelle
aarbelle
MDQ6VXNlcjE0MjQwMzEz
https://api.github.com/users/aarbelle/orgs
https://api.github.com/users/aarbelle/received_events
https://api.github.com/users/aarbelle/repos
https://api.github.com/users/aarbelle/starred{/owner}{/repo}
https://api.github.com/users/aarbelle/subscriptions
User
https://api.github.com/users/aarbelle
public
NONE
@lhoestq Would this work either with an Azure Blob Storage Container or its respective Azure Machine Learning Datastore? If yes, what would that look like in code? I've tried a couple of combinations but no success so far, on the latest version of `datasets`. I need to migrate a dataset to the Azure cloud, `load_dataset("path_to_data")` worked perfectly while the files were local only. Thank you!

@mayorblock would you mind sharing how you got it to work? What did you pass as `storage_options`? Would it maybe work without a custom data loader script?
2023-11-28T22:10:26Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1830839817
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5tIGYJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1830839817/reactions
2023-11-29T09:26:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1830839817
https://avatars.githubusercontent.com/u/17254090?v=4
https://api.github.com/users/csanadpoda/events{/privacy}
https://api.github.com/users/csanadpoda/followers
https://api.github.com/users/csanadpoda/following{/other_user}
https://api.github.com/users/csanadpoda/gists{/gist_id}
https://github.com/csanadpoda
csanadpoda
MDQ6VXNlcjE3MjU0MDkw
https://api.github.com/users/csanadpoda/orgs
https://api.github.com/users/csanadpoda/received_events
https://api.github.com/users/csanadpoda/repos
https://api.github.com/users/csanadpoda/starred{/owner}{/repo}
https://api.github.com/users/csanadpoda/subscriptions
User
https://api.github.com/users/csanadpoda
public
NONE
This ticket would be of so much help.
2023-12-01T16:29:59Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1836419992
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5tdYuY
https://api.github.com/repos/huggingface/datasets/issues/comments/1836419992/reactions
2023-12-01T16:29:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1836419992
https://avatars.githubusercontent.com/u/19391459?v=4
https://api.github.com/users/Sritharan-racap/events{/privacy}
https://api.github.com/users/Sritharan-racap/followers
https://api.github.com/users/Sritharan-racap/following{/other_user}
https://api.github.com/users/Sritharan-racap/gists{/gist_id}
https://github.com/Sritharan-racap
Sritharan-racap
MDQ6VXNlcjE5MzkxNDU5
https://api.github.com/users/Sritharan-racap/orgs
https://api.github.com/users/Sritharan-racap/received_events
https://api.github.com/users/Sritharan-racap/repos
https://api.github.com/users/Sritharan-racap/starred{/owner}{/repo}
https://api.github.com/users/Sritharan-racap/subscriptions
User
https://api.github.com/users/Sritharan-racap
public
CONTRIBUTOR
@lhoestq I've been using this feature for the last year on GCS without problem, but I think we need to fix an issue with S3 and then document the supported calling patterns to reduce confusion

It looks like `datasets` uses a default `DownloadConfig` which is where some potentially unintended storage options are getting passed to fsspec

```
DownloadConfig(
	cache_dir=None, 
	force_download=False, 
	resume_download=False, 
	local_files_only=False, 
	proxies=None, 
	user_agent=None, 
	extract_compressed_file=False, 
	force_extract=False, 
	delete_extracted=False, 
	use_etag=True, 
	num_proc=None, 
	max_retries=1, 
	token=None, 
	ignore_url_params=False, 
	storage_options={'hf': {'token': None, 'endpoint': 'https://huggingface.co'}}, 
	download_desc=None
)
```
(specifically the `storage_options={'hf': {'token': None, 'endpoint': 'https://huggingface.co'}}` part)

`gcsfs` is robust to the extra key in storage options for whatever reason, but `s3fs` is not (haven't dug into why). I'm unable to test `adlfs` but it looks like people here got it working

Is this an issue that needs fixed with s3fs? Or can we avoid passing these default storage options in some cases?

Update: I think probably https://github.com/huggingface/datasets/pull/6127 is where these default storage options were introduced
2024-01-15T21:00:06Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1892759656
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5w0Tho
https://api.github.com/repos/huggingface/datasets/issues/comments/1892759656/reactions
2024-01-15T21:22:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1892759656
https://avatars.githubusercontent.com/u/2512762?v=4
https://api.github.com/users/dwyatte/events{/privacy}
https://api.github.com/users/dwyatte/followers
https://api.github.com/users/dwyatte/following{/other_user}
https://api.github.com/users/dwyatte/gists{/gist_id}
https://github.com/dwyatte
dwyatte
MDQ6VXNlcjI1MTI3NjI=
https://api.github.com/users/dwyatte/orgs
https://api.github.com/users/dwyatte/received_events
https://api.github.com/users/dwyatte/repos
https://api.github.com/users/dwyatte/starred{/owner}{/repo}
https://api.github.com/users/dwyatte/subscriptions
User
https://api.github.com/users/dwyatte
public
MEMBER
Hmm not sure, maybe it has to do with `_prepare_single_hop_path_and_storage_options` returning the "hf" storage options when it shouldn't
2024-01-16T15:36:18Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1893993688
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps5w5AzY
https://api.github.com/repos/huggingface/datasets/issues/comments/1893993688/reactions
2024-01-16T15:36:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1893993688
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Also running into this issue downloading a parquet dataset from S3 (Upload worked fine using [current main branch](https://github.com/huggingface/datasets/commit/90b896193a0a315789022580eb4c80305d168d4d)).
`dataset = Dataset.from_parquet('s3://path-to-file')`
raises 
`TypeError: AioSession.__init__() got an unexpected keyword argument 'hf'`
Found that the issue is introduced in [#6028](https://github.com/huggingface/datasets/commit/14f6edd9222e577dccb962ed5338b79b73502fa5#diff-8a868b8c1127c5cbe7870bbeec30bd2cbec6696a378e65b66782a6935e3f412e)

When commenting out the __post_init__ part to set 'hf', I am able to download the dataset.
2024-03-11T15:38:12Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-1988739257
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps52icC5
https://api.github.com/repos/huggingface/datasets/issues/comments/1988739257/reactions
2024-03-11T15:38:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1988739257
https://avatars.githubusercontent.com/u/1473213?v=4
https://api.github.com/users/Crashkurs/events{/privacy}
https://api.github.com/users/Crashkurs/followers
https://api.github.com/users/Crashkurs/following{/other_user}
https://api.github.com/users/Crashkurs/gists{/gist_id}
https://github.com/Crashkurs
Crashkurs
MDQ6VXNlcjE0NzMyMTM=
https://api.github.com/users/Crashkurs/orgs
https://api.github.com/users/Crashkurs/received_events
https://api.github.com/users/Crashkurs/repos
https://api.github.com/users/Crashkurs/starred{/owner}{/repo}
https://api.github.com/users/Crashkurs/subscriptions
User
https://api.github.com/users/Crashkurs
public
NONE
Can someone paste a complete example for getting this to work? Running this:

```python
from datasets import load_from_disk, load_dataset
from os import environ
from s3fs import S3FileSystem

storage_options = {
    "key": "AzureDiamond",
    "secret": "hunter2",
    "endpoint_url": "https://fly.storage.tigris.dev"
}

model_name = "Qwen/Qwen2.5-32B"
dataset_name = "mlabonne/FineTome-100k"

dataset = load_dataset(
    f"s3://datnybakfu/model-ready/{model_name}/{dataset_name}",
    storage_options=storage_options,
    filesystem=S3FileSystem(),
    streaming=True,
)
```

I get the following error trace:

<details>
<summary>Stacktrace (folded)</summary

```
---------------------------------------------------------------------------

FileNotFoundError                         Traceback (most recent call last)

[<ipython-input-19-0ae7d5f39c46>](https://localhost:8080/#) in <cell line: 29>()
     27 dataset_name = "mlabonne/FineTome-100k"
     28 
---> 29 dataset = load_dataset(
     30     f"s3://datnybakfu/model-ready/{model_name}/{dataset_name}",
     31     storage_options=storage_options,

2 frames

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, keep_in_memory, save_infos, revision, token, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   2130 
   2131     # Create a dataset builder
-> 2132     builder_instance = load_dataset_builder(
   2133         path=path,
   2134         name=name,

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, storage_options, trust_remote_code, _require_default_config_name, **config_kwargs)
   1851         download_config = download_config.copy() if download_config else DownloadConfig()
   1852         download_config.storage_options.update(storage_options)
-> 1853     dataset_module = dataset_module_factory(
   1854         path,
   1855         revision=revision,

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, cache_dir, trust_remote_code, _require_default_config_name, _require_custom_configs, **download_kwargs)
   1733         )
   1734     else:
-> 1735         raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.")
   1736 
   1737 

FileNotFoundError: Couldn't find any data file at /content/s3:/datnybakfu/model-ready/Qwen/Qwen2.5-32B/mlabonne/FineTome-100k.
```

</details>
2024-11-14T17:27:02Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-2477013087
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps6TpDhf
https://api.github.com/repos/huggingface/datasets/issues/comments/2477013087/reactions
2024-11-14T17:27:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2477013087
https://avatars.githubusercontent.com/u/529003?v=4
https://api.github.com/users/Xe/events{/privacy}
https://api.github.com/users/Xe/followers
https://api.github.com/users/Xe/following{/other_user}
https://api.github.com/users/Xe/gists{/gist_id}
https://github.com/Xe
Xe
MDQ6VXNlcjUyOTAwMw==
https://api.github.com/users/Xe/orgs
https://api.github.com/users/Xe/received_events
https://api.github.com/users/Xe/repos
https://api.github.com/users/Xe/starred{/owner}{/repo}
https://api.github.com/users/Xe/subscriptions
User
https://api.github.com/users/Xe
public
MEMBER
Hi, using `s3://` as first argument in load_dataset directly is not supported. You can pass a local path or a HF repository (in the form `username/dataset_name`)

cc @Wauplin if you have a script to move data from S3 to HF ? I know there is this code adapted from https://github.com/fsspec/filesystem_spec/issues/909 that works well (it creates one commit per file)

```python
from multiprocessing.pool import ThreadPool

import fsspec
from tqdm import tqdm

s3_storage_options = {}  # add S3 credentials here if needed, e.g. {"key": aws_access_key_id, "secret": aws_secret_access_key}
a = fsspec.get_mapper("s3://bucket/dataset_folder", **s3_storage_options)
b = fsspec.get_mapper("hf://datasets/username/dataset_name")

def f(k):
 b[k]=a[k]

with ThreadPool(32) as p:
 keys = [key for key in a.keys() if key and not key.endswith("/")]  # ignore root and directories
 for _ in tqdm(p.imap_unordered(f, keys), total=len(keys)):
   pass
```
2024-11-15T09:46:40Z
https://github.com/huggingface/datasets/issues/5281#issuecomment-2478380543
https://api.github.com/repos/huggingface/datasets/issues/5281
IC_kwDODunzps6TuRX_
https://api.github.com/repos/huggingface/datasets/issues/comments/2478380543/reactions
2024-11-15T09:54:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2478380543
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Would be nice to be able to do

```python
data_files=["s3://..."]  # or gs:// or any cloud storage path
storage_options = {...}
load_dataset(..., data_files=data_files, storage_options=storage_options)
```

The idea would be to use `fsspec` as in `download_and_prepare` and `save_to_disk`.

This has been requested several times already. Some users want to use their data from private cloud storage to train models

related:

https://github.com/huggingface/datasets/issues/3490
https://github.com/huggingface/datasets/issues/5244
[forum](https://discuss.huggingface.co/t/how-to-use-s3-path-with-load-dataset-with-streaming-true/25739/2)