Enable parallel shard upload in push_to_hub() using num_proc
MEMBER
it was already added in https://github.com/huggingface/datasets/pull/7606 actually ^^'
2025-07-07T09:31:13Z
https://github.com/huggingface/datasets/pull/7649#issuecomment-3044173910
https://api.github.com/repos/huggingface/datasets/issues/7649
IC_kwDODunzps61cmhW
https://api.github.com/repos/huggingface/datasets/issues/comments/3044173910/reactions
2025-07-07T09:31:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3044173910
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Oh sure sure, Closing this one as redundant.
2025-07-07T18:13:52Z
https://github.com/huggingface/datasets/pull/7649#issuecomment-3046117667
https://api.github.com/repos/huggingface/datasets/issues/7649
IC_kwDODunzps61kBEj
https://api.github.com/repos/huggingface/datasets/issues/comments/3046117667/reactions
2025-07-07T18:13:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3046117667
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
Fixes #7591

### Add num_proc support to `push_to_hub()` for parallel shard upload

This PR adds support for parallel upload of dataset shards via the `num_proc` argument in `Dataset.push_to_hub()`.

📌 While the `num_proc` parameter was already present in the `push_to_hub()` signature and correctly passed to `_push_parquet_shards_to_hub()`, it was not being used to parallelize the upload.

🔧 This PR updates the internal `_push_parquet_shards_to_hub()` function to:

- Use `multiprocessing.Pool` and `iflatmap_unordered()` for concurrent shard upload when `num_proc > 1`
- Preserve original serial upload behavior if `num_proc` is `None` or ≤ 1
- Keep tqdm progress and commit behavior unchanged

Let me know if any test coverage or further changes are needed!
