`datasets.map` multi processing much slower than single processing 
CONTRIBUTOR
Hi @hwijeen, you might want to look at issues #1796 and #1949. I think it could be something related to the I/O operations being performed.
2021-03-05T03:27:51Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-791124517
https://api.github.com/repos/huggingface/datasets/issues/1992
MDEyOklzc3VlQ29tbWVudDc5MTEyNDUxNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/791124517/reactions
2021-03-05T03:27:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791124517
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
NONE
I see that many people are experiencing the same issue. Is this problem considered an "official" bug that is worth a closer look? @lhoestq
2021-03-18T11:28:57Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-801845119
https://api.github.com/repos/huggingface/datasets/issues/1992
MDEyOklzc3VlQ29tbWVudDgwMTg0NTExOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/801845119/reactions
2021-03-18T11:29:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/801845119
https://avatars.githubusercontent.com/u/29157715?v=4
https://api.github.com/users/hwijeen/events{/privacy}
https://api.github.com/users/hwijeen/followers
https://api.github.com/users/hwijeen/following{/other_user}
https://api.github.com/users/hwijeen/gists{/gist_id}
https://github.com/hwijeen
hwijeen
MDQ6VXNlcjI5MTU3NzE1
https://api.github.com/users/hwijeen/orgs
https://api.github.com/users/hwijeen/received_events
https://api.github.com/users/hwijeen/repos
https://api.github.com/users/hwijeen/starred{/owner}{/repo}
https://api.github.com/users/hwijeen/subscriptions
User
https://api.github.com/users/hwijeen
public
MEMBER
Yes this looks like a bug. On my side I haven't managed to reproduce it but @theo-m has. We'll investigate this !
2021-03-18T21:55:37Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-802330624
https://api.github.com/repos/huggingface/datasets/issues/1992
MDEyOklzc3VlQ29tbWVudDgwMjMzMDYyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/802330624/reactions
2023-06-08T12:30:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/802330624
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thank you for the reply! I would be happy to follow the discussions related to the issue.
If you do not mind, could you also give a little more explanation on my p.s.2? I am having a hard time figuring out why the single processing `map` uses all of my cores.
@lhoestq @theo-m 
2021-03-19T00:52:07Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-802423832
https://api.github.com/repos/huggingface/datasets/issues/1992
MDEyOklzc3VlQ29tbWVudDgwMjQyMzgzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/802423832/reactions
2021-03-19T00:52:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/802423832
https://avatars.githubusercontent.com/u/29157715?v=4
https://api.github.com/users/hwijeen/events{/privacy}
https://api.github.com/users/hwijeen/followers
https://api.github.com/users/hwijeen/following{/other_user}
https://api.github.com/users/hwijeen/gists{/gist_id}
https://github.com/hwijeen
hwijeen
MDQ6VXNlcjI5MTU3NzE1
https://api.github.com/users/hwijeen/orgs
https://api.github.com/users/hwijeen/received_events
https://api.github.com/users/hwijeen/repos
https://api.github.com/users/hwijeen/starred{/owner}{/repo}
https://api.github.com/users/hwijeen/subscriptions
User
https://api.github.com/users/hwijeen
public
MEMBER
Regarding your ps2: It depends what function you pass to `map`.
For example, fast tokenizers from `transformers` in Rust tokenize texts and parallelize the tokenization over all the cores.
2021-03-19T11:25:47Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-802763703
https://api.github.com/repos/huggingface/datasets/issues/1992
MDEyOklzc3VlQ29tbWVudDgwMjc2MzcwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/802763703/reactions
2021-03-19T11:25:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/802763703
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I am still experiencing this issue with datasets 1.9.0..
Has there been a further investigation? 
<img width="442" alt="image" src="https://user-images.githubusercontent.com/29157715/126143387-8b5ddca2-a896-4e18-abf7-4fbf62a48b41.png">

2021-07-19T10:03:44Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-882419056
https://api.github.com/repos/huggingface/datasets/issues/1992
IC_kwDODunzps40mKVw
https://api.github.com/repos/huggingface/datasets/issues/comments/882419056/reactions
2021-07-19T10:05:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/882419056
https://avatars.githubusercontent.com/u/29157715?v=4
https://api.github.com/users/hwijeen/events{/privacy}
https://api.github.com/users/hwijeen/followers
https://api.github.com/users/hwijeen/following{/other_user}
https://api.github.com/users/hwijeen/gists{/gist_id}
https://github.com/hwijeen
hwijeen
MDQ6VXNlcjI5MTU3NzE1
https://api.github.com/users/hwijeen/orgs
https://api.github.com/users/hwijeen/received_events
https://api.github.com/users/hwijeen/repos
https://api.github.com/users/hwijeen/starred{/owner}{/repo}
https://api.github.com/users/hwijeen/subscriptions
User
https://api.github.com/users/hwijeen
public
NONE
Hi. Is there any update on this issue? I am desperately trying to decrease my times, and multiprocessing "should" be the solution, but it literally takes 5 times longer.
2022-10-16T22:54:11Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-1280078349
https://api.github.com/repos/huggingface/datasets/issues/1992
IC_kwDODunzps5MTHIN
https://api.github.com/repos/huggingface/datasets/issues/comments/1280078349/reactions
2022-10-16T22:54:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1280078349
https://avatars.githubusercontent.com/u/28630542?v=4
https://api.github.com/users/dduplessis/events{/privacy}
https://api.github.com/users/dduplessis/followers
https://api.github.com/users/dduplessis/following{/other_user}
https://api.github.com/users/dduplessis/gists{/gist_id}
https://github.com/dduplessis
dduplessis
MDQ6VXNlcjI4NjMwNTQy
https://api.github.com/users/dduplessis/orgs
https://api.github.com/users/dduplessis/received_events
https://api.github.com/users/dduplessis/repos
https://api.github.com/users/dduplessis/starred{/owner}{/repo}
https://api.github.com/users/dduplessis/subscriptions
User
https://api.github.com/users/dduplessis
public
MEMBER
Which version of `datasets` are you using ?
2022-10-20T16:28:38Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-1285840703
https://api.github.com/repos/huggingface/datasets/issues/1992
IC_kwDODunzps5MpF8_
https://api.github.com/repos/huggingface/datasets/issues/comments/1285840703/reactions
2022-10-20T16:28:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1285840703
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi,

I’m running into the same issue and trying to come up with a simple benchmark. 

# environment info
I have a total of 80 CPUs.

- `datasets` version: 2.4.0
- Platform: Linux-4.18.0-305.65.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

# How to reproduce

```py
In [1]: from datasets import Dataset, set_caching_enabled                                                                                                       
In [2]: import numpy as np              
In [3]: set_caching_enabled(False)                                                                                                    
In [4]: d = Dataset.from_dict({'foo': np.random.randn(1000,256)})                                                                           
In [9]: d.set_format('np')
In [14]: def sort(array):                                                                                                                   
    ...:     np.sort(array)        
# multiprocessing disabled
In [19]: %%timeit                                                                                                                           
    ...: d.map(sort, input_columns='foo')                                                                                                   
78.8 ms ± 1.22 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)    
# multiprocessing enabled   
In [27]: %%timeit                                                                                                                           
    ...: d.map(sort, input_columns='foo',num_proc=10)               
858 ms ± 45.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)                                                                         
```
2022-11-23T15:51:02Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-1325290638
https://api.github.com/repos/huggingface/datasets/issues/1992
IC_kwDODunzps5O_lSO
https://api.github.com/repos/huggingface/datasets/issues/comments/1325290638/reactions
2022-11-23T15:51:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1325290638
https://avatars.githubusercontent.com/u/25532159?v=4
https://api.github.com/users/PaulLerner/events{/privacy}
https://api.github.com/users/PaulLerner/followers
https://api.github.com/users/PaulLerner/following{/other_user}
https://api.github.com/users/PaulLerner/gists{/gist_id}
https://github.com/PaulLerner
PaulLerner
MDQ6VXNlcjI1NTMyMTU5
https://api.github.com/users/PaulLerner/orgs
https://api.github.com/users/PaulLerner/received_events
https://api.github.com/users/PaulLerner/repos
https://api.github.com/users/PaulLerner/starred{/owner}{/repo}
https://api.github.com/users/PaulLerner/subscriptions
User
https://api.github.com/users/PaulLerner
public
MEMBER
Spawning multiple processes has an overhead. For small datasets the processing is likely to be faster than spawning the processes and passing the data to them.

Especially since your dataset is in memory: the data has to be copied to the subprocesses.
On the other hand, datasets loaded from disk are much faster to reload from a subprocess thanks to memory mapping.
2022-11-24T11:04:32Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-1326294185
https://api.github.com/repos/huggingface/datasets/issues/1992
IC_kwDODunzps5PDaSp
https://api.github.com/repos/huggingface/datasets/issues/comments/1326294185/reactions
2022-11-24T14:04:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1326294185
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks for the clarifications! 

Indeed, when saving then loading the above dataset to disk, and increasing the number of rows to 10K or 100K, the performance gap narrows.

```py
# with 10000 rows
In [3]: %%timeit
   ...: d.map(sort, input_columns='foo')
578 ms ± 5.89 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [4]: %%timeit                                                                                                                                                                                                  
   ...: d.map(sort, input_columns='foo',num_proc=10)                                                                                                                                                              
1.06 s ± 47.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# with 100000 rows
In [6]: %%timeit
   ...: d.map(sort, input_columns='foo')
5.8 s ± 25.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [7]: %%timeit
   ...: d.map(sort, input_columns='foo',num_proc=10)
7.23 s ± 154 ms per loop (mean ± std. dev. of 7 runs, 1 loop each
```
2022-11-24T11:35:36Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-1326329277
https://api.github.com/repos/huggingface/datasets/issues/1992
IC_kwDODunzps5PDi29
https://api.github.com/repos/huggingface/datasets/issues/comments/1326329277/reactions
2022-11-24T11:35:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1326329277
https://avatars.githubusercontent.com/u/25532159?v=4
https://api.github.com/users/PaulLerner/events{/privacy}
https://api.github.com/users/PaulLerner/followers
https://api.github.com/users/PaulLerner/following{/other_user}
https://api.github.com/users/PaulLerner/gists{/gist_id}
https://github.com/PaulLerner
PaulLerner
MDQ6VXNlcjI1NTMyMTU5
https://api.github.com/users/PaulLerner/orgs
https://api.github.com/users/PaulLerner/received_events
https://api.github.com/users/PaulLerner/repos
https://api.github.com/users/PaulLerner/starred{/owner}{/repo}
https://api.github.com/users/PaulLerner/subscriptions
User
https://api.github.com/users/PaulLerner
public
NONE
any updates on this issue? 
I'm using `datasets=2.12.0`. Adding `num_proc` to the mapping function makes it at least 5x slower than using a single process.
2023-06-07T18:22:48Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-1581305865
https://api.github.com/repos/huggingface/datasets/issues/1992
IC_kwDODunzps5eQNAJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1581305865/reactions
2023-06-07T18:22:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1581305865
https://avatars.githubusercontent.com/u/28979824?v=4
https://api.github.com/users/hecmay/events{/privacy}
https://api.github.com/users/hecmay/followers
https://api.github.com/users/hecmay/following{/other_user}
https://api.github.com/users/hecmay/gists{/gist_id}
https://github.com/hecmay
hecmay
MDQ6VXNlcjI4OTc5ODI0
https://api.github.com/users/hecmay/orgs
https://api.github.com/users/hecmay/received_events
https://api.github.com/users/hecmay/repos
https://api.github.com/users/hecmay/starred{/owner}{/repo}
https://api.github.com/users/hecmay/subscriptions
User
https://api.github.com/users/hecmay
public
MEMBER
What kind of function are you passing to `map` ? How many CPUs do you have and what did you set for `num_proc` ?
2023-06-08T12:31:55Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-1582499040
https://api.github.com/repos/huggingface/datasets/issues/1992
IC_kwDODunzps5eUwTg
https://api.github.com/repos/huggingface/datasets/issues/comments/1582499040/reactions
2023-06-08T12:31:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1582499040
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hello! Any solution for this? Thanks!
2024-06-08T20:18:02Z
https://github.com/huggingface/datasets/issues/1992#issuecomment-2156169982
https://api.github.com/repos/huggingface/datasets/issues/1992
IC_kwDODunzps6AhIr-
https://api.github.com/repos/huggingface/datasets/issues/comments/2156169982/reactions
2024-06-08T20:18:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2156169982
https://avatars.githubusercontent.com/u/48152713?v=4
https://api.github.com/users/ymoslem/events{/privacy}
https://api.github.com/users/ymoslem/followers
https://api.github.com/users/ymoslem/following{/other_user}
https://api.github.com/users/ymoslem/gists{/gist_id}
https://github.com/ymoslem
ymoslem
MDQ6VXNlcjQ4MTUyNzEz
https://api.github.com/users/ymoslem/orgs
https://api.github.com/users/ymoslem/received_events
https://api.github.com/users/ymoslem/repos
https://api.github.com/users/ymoslem/starred{/owner}{/repo}
https://api.github.com/users/ymoslem/subscriptions
User
https://api.github.com/users/ymoslem
public
Hi, thank you for the great library.

I've been using datasets to pretrain language models, and it often involves datasets as large as ~70G.
My data preparation step is roughly two steps: `load_dataset` which splits corpora into a table of sentences, and `map` converts a sentence into a list of integers, using a tokenizer.

I noticed that `map` function with `num_proc=mp.cpu_count() //2` takes more than 20 hours to finish the job where as `num_proc=1` gets the job done in about 5 hours. The machine I used has 40 cores, with 126G of RAM. There were no other jobs when `map` function was running.

What could be the reason? I would be happy to provide information necessary to spot the reason.

p.s. I was experiencing the imbalance issue mentioned in [here](https://github.com/huggingface/datasets/issues/610#issuecomment-705177036) when I was using multi processing.
p.s.2 When I run `map` with `num_proc=1`, I see one tqdm bar but all the cores are working. When `num_proc=20`, only 20 cores work. 
![Screen Shot 2021-03-05 at 11 04 59](https://user-images.githubusercontent.com/29157715/110056895-ef6cf000-7da2-11eb-8307-6698e9fb1ad4.png)
