[FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter
MEMBER
Yes that's definitely something we plan to add ^^
2020-07-23T07:46:45Z
https://github.com/huggingface/datasets/issues/426#issuecomment-662866266
https://api.github.com/repos/huggingface/datasets/issues/426
MDEyOklzc3VlQ29tbWVudDY2Mjg2NjI2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/662866266/reactions
2020-07-23T07:46:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/662866266
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Yes, that would be nice. We could take a look at what tensorflow `tf.data` does under the hood for instance.
2020-07-23T08:09:13Z
https://github.com/huggingface/datasets/issues/426#issuecomment-662875243
https://api.github.com/repos/huggingface/datasets/issues/426
MDEyOklzc3VlQ29tbWVudDY2Mjg3NTI0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/662875243/reactions
2020-07-23T08:09:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/662875243
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
So `tf.data.Dataset.map()` returns a `ParallelMapDataset` if `num_parallel_calls is not None` [link](https://github.com/tensorflow/tensorflow/blob/2b96f3662bd776e277f86997659e61046b56c315/tensorflow/python/data/ops/dataset_ops.py#L1623).

There, `num_parallel_calls` is turned into a tensor and and fed to `gen_dataset_ops.parallel_map_dataset` where it looks like tensorflow takes over.

We could start with something simple like a thread or process pool that `imap`s over some shards.
 
2020-07-25T03:15:45Z
https://github.com/huggingface/datasets/issues/426#issuecomment-663800446
https://api.github.com/repos/huggingface/datasets/issues/426
MDEyOklzc3VlQ29tbWVudDY2MzgwMDQ0Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/663800446/reactions
2020-07-25T03:15:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/663800446
https://avatars.githubusercontent.com/u/2000204?v=4
https://api.github.com/users/timothyjlaurent/events{/privacy}
https://api.github.com/users/timothyjlaurent/followers
https://api.github.com/users/timothyjlaurent/following{/other_user}
https://api.github.com/users/timothyjlaurent/gists{/gist_id}
https://github.com/timothyjlaurent
timothyjlaurent
MDQ6VXNlcjIwMDAyMDQ=
https://api.github.com/users/timothyjlaurent/orgs
https://api.github.com/users/timothyjlaurent/received_events
https://api.github.com/users/timothyjlaurent/repos
https://api.github.com/users/timothyjlaurent/starred{/owner}{/repo}
https://api.github.com/users/timothyjlaurent/subscriptions
User
https://api.github.com/users/timothyjlaurent
public
MEMBER
Multiprocessing was added in #552 . You can set the number of processes with `.map(..., num_proc=...)`. It also works for `filter`

Closing this one, but feel free to reo-open if you have other questions
2020-09-07T14:48:04Z
https://github.com/huggingface/datasets/issues/426#issuecomment-688372050
https://api.github.com/repos/huggingface/datasets/issues/426
MDEyOklzc3VlQ29tbWVudDY4ODM3MjA1MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/688372050/reactions
2020-09-07T14:48:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688372050
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq Great feature implemented! Do you have plans to add it to official tutorials [Processing data in a Dataset](https://huggingface.co/docs/datasets/processing.html?highlight=save#augmenting-the-dataset)? It took me sometime to find this parallel processing api.
2021-03-11T02:29:00Z
https://github.com/huggingface/datasets/issues/426#issuecomment-796384823
https://api.github.com/repos/huggingface/datasets/issues/426
MDEyOklzc3VlQ29tbWVudDc5NjM4NDgyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/796384823/reactions
2021-03-11T02:29:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/796384823
https://avatars.githubusercontent.com/u/21215451?v=4
https://api.github.com/users/Chen-Hailin/events{/privacy}
https://api.github.com/users/Chen-Hailin/followers
https://api.github.com/users/Chen-Hailin/following{/other_user}
https://api.github.com/users/Chen-Hailin/gists{/gist_id}
https://github.com/Chen-Hailin
Chen-Hailin
MDQ6VXNlcjIxMjE1NDUx
https://api.github.com/users/Chen-Hailin/orgs
https://api.github.com/users/Chen-Hailin/received_events
https://api.github.com/users/Chen-Hailin/repos
https://api.github.com/users/Chen-Hailin/starred{/owner}{/repo}
https://api.github.com/users/Chen-Hailin/subscriptions
User
https://api.github.com/users/Chen-Hailin
public
MEMBER
Thanks for the heads up !

I just added a paragraph about multiprocessing:
https://huggingface.co/docs/datasets/master/processing.html#multiprocessing
2021-03-12T09:34:12Z
https://github.com/huggingface/datasets/issues/426#issuecomment-797364737
https://api.github.com/repos/huggingface/datasets/issues/426
MDEyOklzc3VlQ29tbWVudDc5NzM2NDczNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/797364737/reactions
2021-03-12T09:34:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797364737
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
It would be nice to be able to speed up `dataset.map` or `dataset.filter`. Perhaps this is as easy as sharding the dataset sending each shard to a process/thread/dask pool and using the new `nlp.concatenate_dataset()` function to join them all together?