storage_options provided to load_dataset not fully piping through since datasets 2.14.0
MEMBER
Hi ! Thanks for reporting, I opened a PR to fix this

What filesystem are you using ?
2023-07-26T10:45:54Z
https://github.com/huggingface/datasets/issues/6071#issuecomment-1651534879
https://api.github.com/repos/huggingface/datasets/issues/6071
IC_kwDODunzps5icGwf
https://api.github.com/repos/huggingface/datasets/issues/comments/1651534879/reactions
2023-07-26T10:45:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1651534879
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq ! Thank you so much 🙌 

It's a bit of a custom setup, but in practice I am using a [pyarrow.fs.S3FileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html) (wrapped in a `fsspec.implementations.arrow.ArrowFSWrapper` [to make it](https://arrow.apache.org/docs/python/filesystems.html#using-arrow-filesystems-with-fsspec) `fsspec` compatible). I also register it as an entrypoint with `fsspec` so that it's the one that gets automatically resolved when looking for filesystems for the `s3` protocol

In my case the `storage_option` that seemed not getting piped through was the filesystem's `endpoint_override` that I use in some tests to point at a mock S3 bucket
2023-07-26T11:00:59Z
https://github.com/huggingface/datasets/issues/6071#issuecomment-1651556305
https://api.github.com/repos/huggingface/datasets/issues/6071
IC_kwDODunzps5icL_R
https://api.github.com/repos/huggingface/datasets/issues/comments/1651556305/reactions
2023-07-26T11:04:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1651556305
https://avatars.githubusercontent.com/u/128361578?v=4
https://api.github.com/users/exs-avianello/events{/privacy}
https://api.github.com/users/exs-avianello/followers
https://api.github.com/users/exs-avianello/following{/other_user}
https://api.github.com/users/exs-avianello/gists{/gist_id}
https://github.com/exs-avianello
exs-avianello
U_kgDOB6akag
https://api.github.com/users/exs-avianello/orgs
https://api.github.com/users/exs-avianello/received_events
https://api.github.com/users/exs-avianello/repos
https://api.github.com/users/exs-avianello/starred{/owner}{/repo}
https://api.github.com/users/exs-avianello/subscriptions
User
https://api.github.com/users/exs-avianello
public
### Describe the bug

Since the latest release of `datasets` (`2.14.0`), custom filesystem `storage_options` passed to `load_dataset()` do not seem to propagate through all the way - leading to problems if loading data files that need those options to be set.

I think this is because of the new `_prepare_path_and_storage_options()` (https://github.com/huggingface/datasets/pull/6028), which returns the right `storage_options` to use given a path and a `DownloadConfig` - but which might not be taking into account the extra `storage_options` explicitly provided e.g. through `load_dataset()`

### Steps to reproduce the bug

```python
import fsspec

import pandas as pd
import datasets

# Generate mock parquet file
data_files = "demo.parquet"
pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}).to_parquet(data_files)

_storage_options = {"x": 1, "y": 2}
fs = fsspec.filesystem("file", **_storage_options)

dataset = datasets.load_dataset(
    "parquet",
    data_files=data_files,
    storage_options=fs.storage_options
)
```

Looking at the `storage_options` resolved here:

https://github.com/huggingface/datasets/blob/b0177910b32712f28d147879395e511207e39958/src/datasets/data_files.py#L331

 they end up being `{}`, instead of propagating through the `storage_options` that were provided to `load_dataset` (`fs.storage_options`). As these then get used for the filesystem operation a few lines below

https://github.com/huggingface/datasets/blob/b0177910b32712f28d147879395e511207e39958/src/datasets/data_files.py#L339

the call will fail if the user-provided `storage_options` were needed.

---

A temporary workaround that seemed to work locally to bypass the problem was to bundle a duplicate of the `storage_options` into the `download_config`, so that they make their way all the way to `_prepare_path_and_storage_options()` and get extracted correctly:

```python
dataset = datasets.load_dataset(
        "parquet",
        data_files=data_files,
        storage_options=fs.storage_options,
        download_config=datasets.DownloadConfig(storage_options={fs.protocol: fs.storage_options}),
    )
```

### Expected behavior

`storage_options` provided to `load_dataset` take effect in all backend filesystem operations.

### Environment info

datasets==2.14.0