Use multiprocess from pathos for multiprocessing
MEMBER
We can just install multiprocess actually, I'll change that
2020-09-21T21:38:44Z
https://github.com/huggingface/datasets/pull/656#issuecomment-696392284
https://api.github.com/repos/huggingface/datasets/issues/656
MDEyOklzc3VlQ29tbWVudDY5NjM5MjI4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/696392284/reactions
2020-09-21T21:38:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696392284
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Just an FYI: I remember that I wanted to try pathos a couple of years back and I ran into issues considering cross-platform; the code would just break on Windows. If I can verify this PR by running CPU tests on Windows, let me know!
2020-09-22T08:40:05Z
https://github.com/huggingface/datasets/pull/656#issuecomment-696587031
https://api.github.com/repos/huggingface/datasets/issues/656
MDEyOklzc3VlQ29tbWVudDY5NjU4NzAzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/696587031/reactions
2020-09-22T08:40:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696587031
https://avatars.githubusercontent.com/u/2779410?v=4
https://api.github.com/users/BramVanroy/events{/privacy}
https://api.github.com/users/BramVanroy/followers
https://api.github.com/users/BramVanroy/following{/other_user}
https://api.github.com/users/BramVanroy/gists{/gist_id}
https://github.com/BramVanroy
BramVanroy
MDQ6VXNlcjI3Nzk0MTA=
https://api.github.com/users/BramVanroy/orgs
https://api.github.com/users/BramVanroy/received_events
https://api.github.com/users/BramVanroy/repos
https://api.github.com/users/BramVanroy/starred{/owner}{/repo}
https://api.github.com/users/BramVanroy/subscriptions
User
https://api.github.com/users/BramVanroy
public
MEMBER
That's good to know thanks
I guess we can just wait for #644 to be merged first. I'm working on fixing the tests for windows
2020-09-22T09:01:45Z
https://github.com/huggingface/datasets/pull/656#issuecomment-696597253
https://api.github.com/repos/huggingface/datasets/issues/656
MDEyOklzc3VlQ29tbWVudDY5NjU5NzI1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/696597253/reactions
2020-09-22T09:01:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696597253
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Looks like all the CI jobs on windows passed !
I also tested locally on my windows and it works great :) 

I think this is ready to merge, let me know if you have any remarks @thomwolf @BramVanroy 
2020-09-25T15:58:26Z
https://github.com/huggingface/datasets/pull/656#issuecomment-699011230
https://api.github.com/repos/huggingface/datasets/issues/656
MDEyOklzc3VlQ29tbWVudDY5OTAxMTIzMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/699011230/reactions
2020-09-25T15:58:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/699011230
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
[Multiprocess](https://github.com/uqfoundation/multiprocess) (from the [pathos](https://github.com/uqfoundation/pathos) project) allows to use lambda functions in multiprocessed map.
It was suggested to use it by @kandorm.

We're already using dill which is its only dependency.