Add num_proc parameter to push_to_hub
CONTRIBUTOR
Hi @SwayStar123 

I'd be interested in taking this up. I plan to add a `num_proc` parameter to `push_to_hub()` and use parallel uploads for shards using `concurrent.futures`. Will explore whether `ThreadPoolExecutor` or `ProcessPoolExecutor` is more suitable based on current implementation. Let me know if that sounds good!

2025-06-27T05:33:42Z
https://github.com/huggingface/datasets/issues/7591#issuecomment-3011731197
https://api.github.com/repos/huggingface/datasets/issues/7591
IC_kwDODunzps6zg179
https://api.github.com/repos/huggingface/datasets/issues/comments/3011731197/reactions
2025-06-27T05:33:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3011731197
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
CONTRIBUTOR
Just a quick update — `push_to_hub()` already had the `num_proc` argument in its signature and was correctly passing it internally to `_push_parquet_shards_to_hub()`.

The actual change required was inside `_push_parquet_shards_to_hub()` to enable parallel shard uploads using `multiprocessing` when `num_proc > 1`.

@lhoestq @SwayStar123 
2025-06-27T06:00:23Z
https://github.com/huggingface/datasets/issues/7591#issuecomment-3011780723
https://api.github.com/repos/huggingface/datasets/issues/7591
IC_kwDODunzps6zhCBz
https://api.github.com/repos/huggingface/datasets/issues/comments/3011780723/reactions
2025-06-27T06:02:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3011780723
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
NONE
> Hi @SwayStar123 
> 
> I'd be interested in taking this up. I plan to add a `num_proc` parameter to `push_to_hub()` and use parallel uploads for shards using `concurrent.futures`. Will explore whether `ThreadPoolExecutor` or `ProcessPoolExecutor` is more suitable based on current implementation. Let me know if that sounds good!
> 

Hey thanks for working on it. But I'm not a hf dev so I don't know the best way to do it.
2025-06-27T06:13:54Z
https://github.com/huggingface/datasets/issues/7591#issuecomment-3011803413
https://api.github.com/repos/huggingface/datasets/issues/7591
IC_kwDODunzps6zhHkV
https://api.github.com/repos/huggingface/datasets/issues/comments/3011803413/reactions
2025-06-27T06:13:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3011803413
https://avatars.githubusercontent.com/u/46050679?v=4
https://api.github.com/users/SwayStar123/events{/privacy}
https://api.github.com/users/SwayStar123/followers
https://api.github.com/users/SwayStar123/following{/other_user}
https://api.github.com/users/SwayStar123/gists{/gist_id}
https://github.com/SwayStar123
SwayStar123
MDQ6VXNlcjQ2MDUwNjc5
https://api.github.com/users/SwayStar123/orgs
https://api.github.com/users/SwayStar123/received_events
https://api.github.com/users/SwayStar123/repos
https://api.github.com/users/SwayStar123/starred{/owner}{/repo}
https://api.github.com/users/SwayStar123/subscriptions
User
https://api.github.com/users/SwayStar123
public
### Feature request

A number of processes parameter to the dataset.push_to_hub method

### Motivation

Shards are currently uploaded serially which makes it slow for many shards, uploading can be done in parallel and much faster
