How to enable `.map()` pre-processing pipelines to support multi-node parallelism?
MEMBER
Right now multiprocessing only runs on single node.

However it's probably possible to extend it to support multi nodes. Indeed we're using the `multiprocess` library from the `pathos` project to do multiprocessing in `datasets`, and `pathos` is made to support parallelism on several nodes. More info about pathos [on the pathos repo](https://github.com/uqfoundation/pathos).

If you're familiar with pathos or if you want to give it a try, it could be a nice addition to the library :)
2020-11-12T23:28:27Z
https://github.com/huggingface/datasets/issues/842#issuecomment-726404000
https://api.github.com/repos/huggingface/datasets/issues/842
MDEyOklzc3VlQ29tbWVudDcyNjQwNDAwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/726404000/reactions
2020-11-12T23:28:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/726404000
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Curious to hear if anything on that side changed or if you suggestions to do it changed @lhoestq :)

For our use-case, we are entering the regime where trading a few more instances to save a few days would be nice :)
2022-10-12T15:19:18Z
https://github.com/huggingface/datasets/issues/842#issuecomment-1276353306
https://api.github.com/repos/huggingface/datasets/issues/842
IC_kwDODunzps5ME5sa
https://api.github.com/repos/huggingface/datasets/issues/comments/1276353306/reactions
2022-10-12T15:19:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1276353306
https://avatars.githubusercontent.com/u/16107619?v=4
https://api.github.com/users/VictorSanh/events{/privacy}
https://api.github.com/users/VictorSanh/followers
https://api.github.com/users/VictorSanh/following{/other_user}
https://api.github.com/users/VictorSanh/gists{/gist_id}
https://github.com/VictorSanh
VictorSanh
MDQ6VXNlcjE2MTA3NjE5
https://api.github.com/users/VictorSanh/orgs
https://api.github.com/users/VictorSanh/received_events
https://api.github.com/users/VictorSanh/repos
https://api.github.com/users/VictorSanh/starred{/owner}{/repo}
https://api.github.com/users/VictorSanh/subscriptions
User
https://api.github.com/users/VictorSanh
public
MEMBER
Currently for multi-node setups we're mostly going towards a nice integration with Dask. But I wouldn't exclude exploring `pathos` more at one point
2022-10-12T16:10:51Z
https://github.com/huggingface/datasets/issues/842#issuecomment-1276422531
https://api.github.com/repos/huggingface/datasets/issues/842
IC_kwDODunzps5MFKmD
https://api.github.com/repos/huggingface/datasets/issues/comments/1276422531/reactions
2022-10-12T16:10:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1276422531
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
What is the current status of this feature? Is multi-node parallelism supported for `.map` function?
2025-03-26T08:26:45Z
https://github.com/huggingface/datasets/issues/842#issuecomment-2753578381
https://api.github.com/repos/huggingface/datasets/issues/842
IC_kwDODunzps6kIEWN
https://api.github.com/repos/huggingface/datasets/issues/comments/2753578381/reactions
2025-03-26T08:26:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2753578381
https://avatars.githubusercontent.com/u/22544721?v=4
https://api.github.com/users/3manifold/events{/privacy}
https://api.github.com/users/3manifold/followers
https://api.github.com/users/3manifold/following{/other_user}
https://api.github.com/users/3manifold/gists{/gist_id}
https://github.com/3manifold
3manifold
MDQ6VXNlcjIyNTQ0NzIx
https://api.github.com/users/3manifold/orgs
https://api.github.com/users/3manifold/received_events
https://api.github.com/users/3manifold/repos
https://api.github.com/users/3manifold/starred{/owner}{/repo}
https://api.github.com/users/3manifold/subscriptions
User
https://api.github.com/users/3manifold
public
MEMBER
Hi ! `datasets` main focus in single node data processing, but you can manually `.shard()` a dataset to run `map()` on each dataset shard in a distributed setup !

Otherwise you can also explore `datatrove` which supports SLURM for distributed data processing, or tools like spark / dask
2025-03-26T09:10:21Z
https://github.com/huggingface/datasets/issues/842#issuecomment-2753682370
https://api.github.com/repos/huggingface/datasets/issues/842
IC_kwDODunzps6kIdvC
https://api.github.com/repos/huggingface/datasets/issues/comments/2753682370/reactions
2025-03-26T09:10:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2753682370
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hi,

Currently, multiprocessing can be enabled for the `.map()` stages on a single node. However, in the case of multi-node training, (since more than one node would be available) I'm wondering if it's possible to extend the parallel processing among nodes, instead of only 1 node running the `.map()` while the other node is waiting for it to finish?

Thanks!