[WIP][Test, Please ignore] Investigate performance impact of using multiprocessing only
COLLABORATOR
`multiprocess` uses `dill` instead of `pickle` for pickling shared objects and, as such, can pickle more types than `multiprocessing`. And I don't think this is something we want to change :).
2023-04-04T13:38:08Z
https://github.com/huggingface/datasets/pull/5703#issuecomment-1495993016
https://api.github.com/repos/huggingface/datasets/issues/5703
IC_kwDODunzps5ZKwq4
https://api.github.com/repos/huggingface/datasets/issues/comments/1495993016/reactions
2023-04-04T13:38:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1495993016
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
That makes sense to me, and I don't think you should merge this change. I was only curious about the performance impact. I saw the benchmarks that was produced in other PRs, and wanted to get a better understanding of it. I created this PR to see if it got automatically added here.

Is there a way I can generate those benchmarks myself?
2023-04-04T18:48:41Z
https://github.com/huggingface/datasets/pull/5703#issuecomment-1496442356
https://api.github.com/repos/huggingface/datasets/issues/5703
IC_kwDODunzps5ZMeX0
https://api.github.com/repos/huggingface/datasets/issues/comments/1496442356/reactions
2023-04-04T18:48:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1496442356
https://avatars.githubusercontent.com/u/1535968?v=4
https://api.github.com/users/hvaara/events{/privacy}
https://api.github.com/users/hvaara/followers
https://api.github.com/users/hvaara/following{/other_user}
https://api.github.com/users/hvaara/gists{/gist_id}
https://github.com/hvaara
hvaara
MDQ6VXNlcjE1MzU5Njg=
https://api.github.com/users/hvaara/orgs
https://api.github.com/users/hvaara/received_events
https://api.github.com/users/hvaara/repos
https://api.github.com/users/hvaara/starred{/owner}{/repo}
https://api.github.com/users/hvaara/subscriptions
User
https://api.github.com/users/hvaara
public
COLLABORATOR
You can find some speed comparisons between dill and pickle on SO if you google "dill vs pickle speed".

And for the benchmarks, you can generate them locally with DVC running this code from the repo root: https://github.com/huggingface/datasets/blob/0803a006db1c395ac715662cc6079651f77c11ea/.github/workflows/benchmarks.yaml#L23-L47.
2023-04-05T12:59:41Z
https://github.com/huggingface/datasets/pull/5703#issuecomment-1497445967
https://api.github.com/repos/huggingface/datasets/issues/5703
IC_kwDODunzps5ZQTZP
https://api.github.com/repos/huggingface/datasets/issues/comments/1497445967/reactions
2023-04-05T12:59:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497445967
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Thanks for the help @mariosasko!
2023-04-20T03:17:32Z
https://github.com/huggingface/datasets/pull/5703#issuecomment-1515654849
https://api.github.com/repos/huggingface/datasets/issues/5703
IC_kwDODunzps5aVw7B
https://api.github.com/repos/huggingface/datasets/issues/comments/1515654849/reactions
2023-04-20T03:17:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1515654849
https://avatars.githubusercontent.com/u/1535968?v=4
https://api.github.com/users/hvaara/events{/privacy}
https://api.github.com/users/hvaara/followers
https://api.github.com/users/hvaara/following{/other_user}
https://api.github.com/users/hvaara/gists{/gist_id}
https://github.com/hvaara
hvaara
MDQ6VXNlcjE1MzU5Njg=
https://api.github.com/users/hvaara/orgs
https://api.github.com/users/hvaara/received_events
https://api.github.com/users/hvaara/repos
https://api.github.com/users/hvaara/starred{/owner}{/repo}
https://api.github.com/users/hvaara/subscriptions
User
https://api.github.com/users/hvaara
public