Pass new_fingerprint in multiprocessing
NONE
@lhoestq Hi~, does this support that `datasets.map(func, batched=True, batch_size, num_proc>1, new_fingerprint="func_v1")` even if `func` can't pickle. I also notice that you said "Unfortunately you need picklable mapping functions to make multiprocessing work :confused: Also feel free to open an issue or send me a dm if you are in a situation where the caching fails. I can help you with that :slight_smile:" in [here](https://discuss.huggingface.co/t/how-to-deal-with-unpickable-objects-in-map/1547/8). So, I want to ask that is there a way for users to use multiprocessing in `datasets.map` when the `func` can't pickle? 
Thanks in advance!
2022-08-19T09:14:07Z
https://github.com/huggingface/datasets/pull/3409#issuecomment-1220444474
https://api.github.com/repos/huggingface/datasets/issues/3409
IC_kwDODunzps5IvoE6
https://api.github.com/repos/huggingface/datasets/issues/comments/1220444474/reactions
2022-08-19T09:14:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1220444474
https://avatars.githubusercontent.com/u/55944993?v=4
https://api.github.com/users/nxznm/events{/privacy}
https://api.github.com/users/nxznm/followers
https://api.github.com/users/nxznm/following{/other_user}
https://api.github.com/users/nxznm/gists{/gist_id}
https://github.com/nxznm
nxznm
MDQ6VXNlcjU1OTQ0OTkz
https://api.github.com/users/nxznm/orgs
https://api.github.com/users/nxznm/received_events
https://api.github.com/users/nxznm/repos
https://api.github.com/users/nxznm/starred{/owner}{/repo}
https://api.github.com/users/nxznm/subscriptions
User
https://api.github.com/users/nxznm
public
MEMBER
Yea you need your function to be picklable for multiprocessing, otherwise the main process is not able to pass your function to the child processes.
2022-08-19T10:41:04Z
https://github.com/huggingface/datasets/pull/3409#issuecomment-1220525680
https://api.github.com/repos/huggingface/datasets/issues/3409
IC_kwDODunzps5Iv75w
https://api.github.com/repos/huggingface/datasets/issues/comments/1220525680/reactions
2022-08-19T10:41:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1220525680
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Following https://github.com/huggingface/datasets/pull/3045

Currently one can pass `new_fingerprint` to `.map()` to use a custom fingerprint instead of the one computed by hashing the map transform. However it's ignored if `num_proc>1`.

In this PR I fixed that by passing `new_fingerprint` to `._map_single()` when `num_proc>1`.
More specifically, `new_fingerprint` with a suffix based on the process `rank` is passed, so that each process has a different `new_fingerprint`

cc @TevenLeScao @vlievin