Support proxy and user-agent in fsspec calls
NONE
hii @lhoestq can you assign this issue to me?

2023-10-09T12:29:45Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-1752920716
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps5oe3KM
https://api.github.com/repos/huggingface/datasets/issues/comments/1752920716/reactions
2023-10-09T12:29:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1752920716
https://avatars.githubusercontent.com/u/95092167?v=4
https://api.github.com/users/zutarich/events{/privacy}
https://api.github.com/users/zutarich/followers
https://api.github.com/users/zutarich/following{/other_user}
https://api.github.com/users/zutarich/gists{/gist_id}
https://github.com/zutarich
zutarich
U_kgDOBar9xw
https://api.github.com/users/zutarich/orgs
https://api.github.com/users/zutarich/received_events
https://api.github.com/users/zutarich/repos
https://api.github.com/users/zutarich/starred{/owner}{/repo}
https://api.github.com/users/zutarich/subscriptions
User
https://api.github.com/users/zutarich
public
MEMBER
You can reply "#self-assign" to this issue to automatically get assigned to it :)
Let me know if you have any questions or if I can help
2023-10-09T12:53:27Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-1752957449
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps5ofAIJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1752957449/reactions
2023-10-09T12:53:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1752957449
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
#2289 
2023-10-09T12:54:55Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-1752959699
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps5ofArT
https://api.github.com/repos/huggingface/datasets/issues/comments/1752959699/reactions
2023-10-09T12:54:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1752959699
https://avatars.githubusercontent.com/u/95092167?v=4
https://api.github.com/users/zutarich/events{/privacy}
https://api.github.com/users/zutarich/followers
https://api.github.com/users/zutarich/following{/other_user}
https://api.github.com/users/zutarich/gists{/gist_id}
https://github.com/zutarich
zutarich
U_kgDOBar9xw
https://api.github.com/users/zutarich/orgs
https://api.github.com/users/zutarich/received_events
https://api.github.com/users/zutarich/repos
https://api.github.com/users/zutarich/starred{/owner}{/repo}
https://api.github.com/users/zutarich/subscriptions
User
https://api.github.com/users/zutarich
public
NONE
Actually i am quite new to figure it out how everything goes and done 

> You can reply "#self-assign" to this issue to automatically get assigned to it :)
> Let me know if you have any questions or if I can help

when i wrote #self-assign it automatically got converted to some number is it correct or i have done it some wrong way, I am quite new to open source thus wanna try to learn and explore it
2023-10-09T12:57:04Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-1752963408
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps5ofBlQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1752963408/reactions
2023-10-09T12:57:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1752963408
https://avatars.githubusercontent.com/u/95092167?v=4
https://api.github.com/users/zutarich/events{/privacy}
https://api.github.com/users/zutarich/followers
https://api.github.com/users/zutarich/following{/other_user}
https://api.github.com/users/zutarich/gists{/gist_id}
https://github.com/zutarich
zutarich
U_kgDOBar9xw
https://api.github.com/users/zutarich/orgs
https://api.github.com/users/zutarich/received_events
https://api.github.com/users/zutarich/repos
https://api.github.com/users/zutarich/starred{/owner}{/repo}
https://api.github.com/users/zutarich/subscriptions
User
https://api.github.com/users/zutarich
public
NONE
#2289 #self-assign 
2023-10-09T12:57:56Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-1752964877
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps5ofB8N
https://api.github.com/repos/huggingface/datasets/issues/comments/1752964877/reactions
2023-10-09T12:57:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1752964877
https://avatars.githubusercontent.com/u/95092167?v=4
https://api.github.com/users/zutarich/events{/privacy}
https://api.github.com/users/zutarich/followers
https://api.github.com/users/zutarich/following{/other_user}
https://api.github.com/users/zutarich/gists{/gist_id}
https://github.com/zutarich
zutarich
U_kgDOBar9xw
https://api.github.com/users/zutarich/orgs
https://api.github.com/users/zutarich/received_events
https://api.github.com/users/zutarich/repos
https://api.github.com/users/zutarich/starred{/owner}{/repo}
https://api.github.com/users/zutarich/subscriptions
User
https://api.github.com/users/zutarich
public
MEMBER
Ah yea github tries to replace the #self-assign with an issue link. I guess you can try to copy-paste instead to see if it works

Anyway let me assign you manually
2023-10-09T13:00:48Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-1752969329
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps5ofDBx
https://api.github.com/repos/huggingface/datasets/issues/comments/1752969329/reactions
2023-10-09T13:01:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1752969329
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
thanks a lot @lhoestq ! though i have a very lil idea of the issue, i am new. as i said before,  but gonna try my best shot to do it.
can you please suggest some tips or anything from your side, how basically we approach it will be really helpfull.
Will try my best!
2023-10-09T13:21:08Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-1753003515
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps5ofLX7
https://api.github.com/repos/huggingface/datasets/issues/comments/1753003515/reactions
2023-10-09T13:21:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1753003515
https://avatars.githubusercontent.com/u/95092167?v=4
https://api.github.com/users/zutarich/events{/privacy}
https://api.github.com/users/zutarich/followers
https://api.github.com/users/zutarich/following{/other_user}
https://api.github.com/users/zutarich/gists{/gist_id}
https://github.com/zutarich
zutarich
U_kgDOBar9xw
https://api.github.com/users/zutarich/orgs
https://api.github.com/users/zutarich/received_events
https://api.github.com/users/zutarich/repos
https://api.github.com/users/zutarich/starred{/owner}{/repo}
https://api.github.com/users/zutarich/subscriptions
User
https://api.github.com/users/zutarich
public
MEMBER
The HfFileSystem from the `huggingface_hub` package can already read the HTTP_PROXY and HTTPS_PROXY environment variables. So the remaining thing missing is the `user_agent` that the user may include in a `DownloadConfig` object. The user agent can be used for regular http calls but also calls to the HfFileSystem.

- for http, the `user_agent` isn't passed from `DownloadConfig` to `get_datasets_user_agent` in `_prepare_single_hop_path_and_storage_options` in `streaming_download_manager.py` so we need to include it
- for HfFileSystem I think it requires a PR in https://github.com/huggingface/huggingface_hub to include it in the `HfFileSystem.__init__`
2023-10-09T13:49:14Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-1753052447
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps5ofXUf
https://api.github.com/repos/huggingface/datasets/issues/comments/1753052447/reactions
2023-10-09T13:49:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1753052447
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq 👋🏼

Is anyone currently working on this? If not, I'd like to pick it up.

As I understand it:
- The `user_agent` from `DownloadConfig` isn't currently passed to `get_datasets_user_agent()` inside `_prepare_single_hop_path_and_storage_options`.
- I'll update that function to include the correct `user-agent` in the `headers`.
- For full support, we may also need a change in `huggingface_hub` to let `HfFileSystem` accept custom headers.

Please let me know if this approach sounds good or if you’d prefer it handled differently 🙌

2025-06-21T14:06:31Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-2993600735
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps6ybrjf
https://api.github.com/repos/huggingface/datasets/issues/comments/2993600735/reactions
2025-06-21T14:06:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2993600735
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
CONTRIBUTOR
Hi @lhoestq 👋🏼 Just following up on this one!

I’ve opened [PR #7631](https://github.com/huggingface/datasets/pull/7631) to pass the user_agent from DownloadConfig into the fsspec storage options via _prepare_single_hop_path_and_storage_options (for HTTP/S).

For full support in HfFileSystem, I understand we might need a corresponding PR in huggingface_hub to add user_agent handling in its __init__. If you're okay with that direction, I can also raise a PR there!

Let me know if any adjustments are needed 🙌
2025-06-26T18:26:27Z
https://github.com/huggingface/datasets/issues/6046#issuecomment-3009446911
https://api.github.com/repos/huggingface/datasets/issues/6046
IC_kwDODunzps6zYIP_
https://api.github.com/repos/huggingface/datasets/issues/comments/3009446911/reactions
2025-06-26T18:26:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3009446911
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
Since we switched to the new HfFileSystem we no longer apply user's proxy and user-agent.

Using the HTTP_PROXY and HTTPS_PROXY environment variables works though since we use aiohttp to call the HF Hub.

This can be implemented in `_prepare_single_hop_path_and_storage_options`.

Though ideally the `HfFileSystem` could support passing at least the proxies