Unexpected keyword argument 'hf' when downloading CSV dataset from S3
NONE
I am facing similar issue while reading a csv file from s3. Wondering if somebody has found a workaround. 
2024-01-18T00:44:56Z
https://github.com/huggingface/datasets/issues/6598#issuecomment-1897564947
https://api.github.com/repos/huggingface/datasets/issues/6598
IC_kwDODunzps5xGosT
https://api.github.com/repos/huggingface/datasets/issues/comments/1897564947/reactions
2024-01-18T00:44:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1897564947
https://avatars.githubusercontent.com/u/156721393?v=4
https://api.github.com/users/deepbot86/events{/privacy}
https://api.github.com/users/deepbot86/followers
https://api.github.com/users/deepbot86/following{/other_user}
https://api.github.com/users/deepbot86/gists{/gist_id}
https://github.com/deepbot86
deepbot86
U_kgDOCVdg8Q
https://api.github.com/users/deepbot86/orgs
https://api.github.com/users/deepbot86/received_events
https://api.github.com/users/deepbot86/repos
https://api.github.com/users/deepbot86/starred{/owner}{/repo}
https://api.github.com/users/deepbot86/subscriptions
User
https://api.github.com/users/deepbot86
public
NONE
same thing happened to other formats like parquet
2024-01-26T22:53:33Z
https://github.com/huggingface/datasets/issues/6598#issuecomment-1912799101
https://api.github.com/repos/huggingface/datasets/issues/6598
IC_kwDODunzps5yAv99
https://api.github.com/repos/huggingface/datasets/issues/comments/1912799101/reactions
2024-01-26T22:53:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1912799101
https://avatars.githubusercontent.com/u/35147961?v=4
https://api.github.com/users/ChenchaoZhao/events{/privacy}
https://api.github.com/users/ChenchaoZhao/followers
https://api.github.com/users/ChenchaoZhao/following{/other_user}
https://api.github.com/users/ChenchaoZhao/gists{/gist_id}
https://github.com/ChenchaoZhao
ChenchaoZhao
MDQ6VXNlcjM1MTQ3OTYx
https://api.github.com/users/ChenchaoZhao/orgs
https://api.github.com/users/ChenchaoZhao/received_events
https://api.github.com/users/ChenchaoZhao/repos
https://api.github.com/users/ChenchaoZhao/starred{/owner}{/repo}
https://api.github.com/users/ChenchaoZhao/subscriptions
User
https://api.github.com/users/ChenchaoZhao
public
NONE
I am facing similar issue while reading a parquet file from s3.
i try with every version between 2.14 to 2.16.1 but it dosen't  work 
2024-02-04T10:55:26Z
https://github.com/huggingface/datasets/issues/6598#issuecomment-1925701157
https://api.github.com/repos/huggingface/datasets/issues/6598
IC_kwDODunzps5yx94l
https://api.github.com/repos/huggingface/datasets/issues/comments/1925701157/reactions
2024-02-04T10:55:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1925701157
https://avatars.githubusercontent.com/u/60541252?v=4
https://api.github.com/users/SinaTavakoli/events{/privacy}
https://api.github.com/users/SinaTavakoli/followers
https://api.github.com/users/SinaTavakoli/following{/other_user}
https://api.github.com/users/SinaTavakoli/gists{/gist_id}
https://github.com/SinaTavakoli
SinaTavakoli
MDQ6VXNlcjYwNTQxMjUy
https://api.github.com/users/SinaTavakoli/orgs
https://api.github.com/users/SinaTavakoli/received_events
https://api.github.com/users/SinaTavakoli/repos
https://api.github.com/users/SinaTavakoli/starred{/owner}{/repo}
https://api.github.com/users/SinaTavakoli/subscriptions
User
https://api.github.com/users/SinaTavakoli
public
NONE
Re-define the DownloadConfig might work:

```
class ReviseDownloadConfig(DownloadConfig):
    def __post_init__(self, use_auth_token):
        if use_auth_token != "deprecated":
            warnings.warn(
                "'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n"
                f"You can remove this warning by passing 'token={use_auth_token}' instead.",
                FutureWarning,
            )
            self.token = use_auth_token

    def copy(self):
        return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})

downloadconfig = ReviseDownloadConfig()
```

2024-02-05T05:42:58Z
https://github.com/huggingface/datasets/issues/6598#issuecomment-1926271027
https://api.github.com/repos/huggingface/datasets/issues/6598
IC_kwDODunzps5y0JAz
https://api.github.com/repos/huggingface/datasets/issues/comments/1926271027/reactions
2024-02-05T05:44:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1926271027
https://avatars.githubusercontent.com/u/14846358?v=4
https://api.github.com/users/pandaczm/events{/privacy}
https://api.github.com/users/pandaczm/followers
https://api.github.com/users/pandaczm/following{/other_user}
https://api.github.com/users/pandaczm/gists{/gist_id}
https://github.com/pandaczm
pandaczm
MDQ6VXNlcjE0ODQ2MzU4
https://api.github.com/users/pandaczm/orgs
https://api.github.com/users/pandaczm/received_events
https://api.github.com/users/pandaczm/repos
https://api.github.com/users/pandaczm/starred{/owner}{/repo}
https://api.github.com/users/pandaczm/subscriptions
User
https://api.github.com/users/pandaczm
public
NONE
> Re-define the DownloadConfig might work:
> 
> ```
> class ReviseDownloadConfig(DownloadConfig):
>     def __post_init__(self, use_auth_token):
>         if use_auth_token != "deprecated":
>             warnings.warn(
>                 "'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n"
>                 f"You can remove this warning by passing 'token={use_auth_token}' instead.",
>                 FutureWarning,
>             )
>             self.token = use_auth_token
> ```
This seemed to work for me.

2024-03-09T02:32:00Z
https://github.com/huggingface/datasets/issues/6598#issuecomment-1986699619
https://api.github.com/repos/huggingface/datasets/issues/6598
IC_kwDODunzps52aqFj
https://api.github.com/repos/huggingface/datasets/issues/comments/1986699619/reactions
2024-03-09T02:32:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1986699619
https://avatars.githubusercontent.com/u/159478?v=4
https://api.github.com/users/charlescearl/events{/privacy}
https://api.github.com/users/charlescearl/followers
https://api.github.com/users/charlescearl/following{/other_user}
https://api.github.com/users/charlescearl/gists{/gist_id}
https://github.com/charlescearl
charlescearl
MDQ6VXNlcjE1OTQ3OA==
https://api.github.com/users/charlescearl/orgs
https://api.github.com/users/charlescearl/received_events
https://api.github.com/users/charlescearl/repos
https://api.github.com/users/charlescearl/starred{/owner}{/repo}
https://api.github.com/users/charlescearl/subscriptions
User
https://api.github.com/users/charlescearl
public
NONE
use pandas and then convert to `Dataset`
2024-03-22T02:08:05Z
https://github.com/huggingface/datasets/issues/6598#issuecomment-2014182560
https://api.github.com/repos/huggingface/datasets/issues/6598
IC_kwDODunzps54Dfyg
https://api.github.com/repos/huggingface/datasets/issues/comments/2014182560/reactions
2024-03-22T02:08:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2014182560
https://avatars.githubusercontent.com/u/35147961?v=4
https://api.github.com/users/ChenchaoZhao/events{/privacy}
https://api.github.com/users/ChenchaoZhao/followers
https://api.github.com/users/ChenchaoZhao/following{/other_user}
https://api.github.com/users/ChenchaoZhao/gists{/gist_id}
https://github.com/ChenchaoZhao
ChenchaoZhao
MDQ6VXNlcjM1MTQ3OTYx
https://api.github.com/users/ChenchaoZhao/orgs
https://api.github.com/users/ChenchaoZhao/received_events
https://api.github.com/users/ChenchaoZhao/repos
https://api.github.com/users/ChenchaoZhao/starred{/owner}{/repo}
https://api.github.com/users/ChenchaoZhao/subscriptions
User
https://api.github.com/users/ChenchaoZhao
public
NONE
I am currently facing the same issue while using a custom loading script with files located in a remote S3 instance. I was using the `download_custom` functionality but now it is deprecated mentioning that I should use the native S3 loading, which is not working. 

As stated before, the library forces the existence of a `hf` key in the `storage_options` variable, which is **not** accepted by  `s3fs` : 

```python
.../site-packages/s3fs/core.py", line 516, in set_session
    self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: __init__() got an unexpected keyword argument 'hf'.
````

Meanwhile, if my `storage_options` var stays like:
```python
{'key': '...',
 'secret': '...',
 'client_kwargs': {'endpoint_url': '...'}}
```
it works alright. 
2024-04-24T13:41:34Z
https://github.com/huggingface/datasets/issues/6598#issuecomment-2074981667
https://api.github.com/repos/huggingface/datasets/issues/6598
IC_kwDODunzps57rbUj
https://api.github.com/repos/huggingface/datasets/issues/comments/2074981667/reactions
2024-04-24T13:43:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2074981667
https://avatars.githubusercontent.com/u/1085210?v=4
https://api.github.com/users/psorianom/events{/privacy}
https://api.github.com/users/psorianom/followers
https://api.github.com/users/psorianom/following{/other_user}
https://api.github.com/users/psorianom/gists{/gist_id}
https://github.com/psorianom
psorianom
MDQ6VXNlcjEwODUyMTA=
https://api.github.com/users/psorianom/orgs
https://api.github.com/users/psorianom/received_events
https://api.github.com/users/psorianom/repos
https://api.github.com/users/psorianom/starred{/owner}{/repo}
https://api.github.com/users/psorianom/subscriptions
User
https://api.github.com/users/psorianom
public
NONE
Did anyone look into similar issues with model upload? setting s3 for checkpointing return `FileNotFoundError`
2025-01-31T15:35:31Z
https://github.com/huggingface/datasets/issues/6598#issuecomment-2627635398
https://api.github.com/repos/huggingface/datasets/issues/6598
IC_kwDODunzps6cnojG
https://api.github.com/repos/huggingface/datasets/issues/comments/2627635398/reactions
2025-01-31T15:35:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2627635398
https://avatars.githubusercontent.com/u/46866776?v=4
https://api.github.com/users/nicolaiberk/events{/privacy}
https://api.github.com/users/nicolaiberk/followers
https://api.github.com/users/nicolaiberk/following{/other_user}
https://api.github.com/users/nicolaiberk/gists{/gist_id}
https://github.com/nicolaiberk
nicolaiberk
MDQ6VXNlcjQ2ODY2Nzc2
https://api.github.com/users/nicolaiberk/orgs
https://api.github.com/users/nicolaiberk/received_events
https://api.github.com/users/nicolaiberk/repos
https://api.github.com/users/nicolaiberk/starred{/owner}{/repo}
https://api.github.com/users/nicolaiberk/subscriptions
User
https://api.github.com/users/nicolaiberk
public
### Describe the bug

I receive this error message when using `load_dataset` with "csv" path and `dataset_files=s3://...`:
```
TypeError: Session.__init__() got an unexpected keyword argument 'hf'
```

I found a similar issue here: https://stackoverflow.com/questions/77596258/aws-issue-load-dataset-from-s3-fails-with-unexpected-keyword-argument-error-in

Full stacktrace:
```
.../site-packages/datasets/load.py:2549: in load_dataset
    builder_instance.download_and_prepare(
.../site-packages/datasets/builder.py:1005: in download_and_prepare
    self._download_and_prepare(
.../site-packages/datasets/builder.py:1078: in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
.../site-packages/datasets/packaged_modules/csv/csv.py:147: in _split_generators
    data_files = dl_manager.download_and_extract(self.config.data_files)
.../site-packages/datasets/download/download_manager.py:562: in download_and_extract
    return self.extract(self.download(url_or_urls))
.../site-packages/datasets/download/download_manager.py:426: in download
    downloaded_path_or_paths = map_nested(
.../site-packages/datasets/utils/py_utils.py:466: in map_nested
    mapped = [
.../site-packages/datasets/utils/py_utils.py:467: in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
.../site-packages/datasets/utils/py_utils.py:387: in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
.../site-packages/datasets/utils/py_utils.py:387: in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
.../site-packages/datasets/utils/py_utils.py:370: in _single_map_nested
    return function(data_struct)
.../site-packages/datasets/download/download_manager.py:451: in _download
    out = cached_path(url_or_filename, download_config=download_config)
.../site-packages/datasets/utils/file_utils.py:188: in cached_path
    output_path = get_from_cache(
...1/site-packages/datasets/utils/file_utils.py:511: in get_from_cache
    response = fsspec_head(url, storage_options=storage_options)
.../site-packages/datasets/utils/file_utils.py:316: in fsspec_head
    fs, _, paths = fsspec.get_fs_token_paths(url, storage_options=storage_options)
.../site-packages/fsspec/core.py:622: in get_fs_token_paths
    fs = filesystem(protocol, **inkwargs)
.../site-packages/fsspec/registry.py:290: in filesystem
    return cls(**storage_options)
.../site-packages/fsspec/spec.py:79: in __call__
    obj = super().__call__(*args, **kwargs)
.../site-packages/s3fs/core.py:187: in __init__
    self.s3 = self.connect()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <s3fs.core.S3FileSystem object at 0x1500a1310>, refresh = True

    def connect(self, refresh=True):
        """
        Establish S3 connection object.
    
        Parameters
        ----------
        refresh : bool
            Whether to create new session/client, even if a previous one with
            the same parameters already exists. If False (default), an
            existing one will be used if possible
        """
        if refresh is False:
            # back compat: we store whole FS instance now
            return self.s3
        anon, key, secret, kwargs, ckwargs, token, ssl = (
            self.anon, self.key, self.secret, self.kwargs,
            self.client_kwargs, self.token, self.use_ssl)
    
        if not self.passed_in_session:
>           self.session = botocore.session.Session(**self.kwargs)
E           TypeError: Session.__init__() got an unexpected keyword argument 'hf'
```

### Steps to reproduce the bug

1. Assuming a valid CSV file located at `s3://bucket/data.csv`
2. Run the below code:
```
storage_options = {
    "key": "...",
    "secret": "...",
    "client_kwargs": {
        "endpoint_url": "...",
    }
}
load_dataset("csv", data_files="s3://bucket/data.csv", storage_options=storage_options)
```

Encountered in version `2.16.1` but also reproduced in `2.16.0` and `2.15.0`.

Note: I encountered this in a unit test using a `moto` mock for S3, however since the error occurs before the session is instantiated, it should not be the issue.

### Expected behavior

No exception is raised, the boto3 session is created successfully, and the CSV file is downloaded successfully and returned as a dataset.

===

After some research I found that `DownloadConfig` has a `__post_init__` method that always forces this value to be set in its `storage_options`, even though in case of an S3 location the storage options get passed on to the S3 Session which does not expect this parameter. I assume this parameter is needed when reading from the huggingface hub and should not be set in this context.

Unfortunately there is nothing the user can do to work around it. Even if you manually do something like:
```
download_config = DownloadConfig()
del download_config.storage_options["hf"]
load_dataset("csv", data_files="s3://bucket/data.csv", download_config=download_config)
```
the library will still reinsert this parameter when `download_config = self.download_config.copy()` in line 418 of `download_manager.py` (`DownloadManager.download`).

Therefore `load_dataset` currently cannot be used to read a dataset in CSV format from an S3 location.

### Environment info

- `datasets` version: 2.16.1
- Platform: macOS-14.2.1-arm64-arm-64bit
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
