Pass user-agent from DownloadConfig into fsspec storage_options
CONTRIBUTOR
- This PR assumes that `HfFileSystem` in `huggingface_hub` supports receiving `headers` in `storage_options`. If not, a follow-up PR can be opened to add this support to `HfFileSystem.__init__`.
- No test was added for this since it’s a config passthrough. If needed, I’d be happy to add one.
2025-06-21T14:25:28Z
https://github.com/huggingface/datasets/pull/7631#issuecomment-2993608438
https://api.github.com/repos/huggingface/datasets/issues/7631
IC_kwDODunzps6ybtb2
https://api.github.com/repos/huggingface/datasets/issues/comments/2993608438/reactions
2025-06-21T14:25:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2993608438
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
Fixes part of issue #6046

### Problem
The `user-agent` defined in `DownloadConfig` was not passed down to fsspec-based filesystems like `HfFileSystem`, which prevents proper identification/tracking of client requests.

### Solution
Added support for injecting the `user-agent` into `storage_options["headers"]` within `_prepare_single_hop_path_and_storage_options()` based on the `protocol`.

Now, when using `hf://`, `http://`, or `https://`, the custom user-agent is passed automatically.

### Code Location
Modified:
- `src/datasets/utils/file_utils.py`

Used `get_datasets_user_agent(...)` to ensure proper formatting and fallback logic.