Adding arg to pass process rank to `map`
CONTRIBUTOR
Some commits seem to be there twice (made the mistake of rebasing because I wasn't sure whether the doc had changed), is this an issue @lhoestq ?
2021-11-23T17:18:55Z
https://github.com/huggingface/datasets/pull/3314#issuecomment-976877711
https://api.github.com/repos/huggingface/datasets/issues/3314
IC_kwDODunzps46OfiP
https://api.github.com/repos/huggingface/datasets/issues/comments/976877711/reactions
2021-11-23T17:18:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/976877711
https://avatars.githubusercontent.com/u/26709476?v=4
https://api.github.com/users/TevenLeScao/events{/privacy}
https://api.github.com/users/TevenLeScao/followers
https://api.github.com/users/TevenLeScao/following{/other_user}
https://api.github.com/users/TevenLeScao/gists{/gist_id}
https://github.com/TevenLeScao
TevenLeScao
MDQ6VXNlcjI2NzA5NDc2
https://api.github.com/users/TevenLeScao/orgs
https://api.github.com/users/TevenLeScao/received_events
https://api.github.com/users/TevenLeScao/repos
https://api.github.com/users/TevenLeScao/starred{/owner}{/repo}
https://api.github.com/users/TevenLeScao/subscriptions
User
https://api.github.com/users/TevenLeScao
public
This PR adds a `with_rank` argument to `map` that gives the user the possibility to pass the rank of each process to their function. This is mostly designed for multi-GPU map (each process can be sent to a different device thanks to the rank). I've also added tests. I'm putting the PR up so you can check the code, I'll add a multi-GPU example to the doc (+ write a bit in the doc for the new arg)