Process 0 very slow when using num_procs with map to tokenizer
MEMBER
Hi ! Thanks for reporting.
Is the distribution of text length of your data evenly distributed across your dataset ? I mean, could it be because the examples in the first part of your dataset are slower to process ?
Also could how many CPUs can you use for multiprocessing ?
```python
import multiprocessing
print(multiprocessing.cpu_count())
```
Which tokenizer are you using ?
2020-10-26T09:20:48Z
https://github.com/huggingface/datasets/issues/758#issuecomment-716420989
https://api.github.com/repos/huggingface/datasets/issues/758
MDEyOklzc3VlQ29tbWVudDcxNjQyMDk4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/716420989/reactions
2020-10-26T09:20:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/716420989
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Using pre trained HF tokenizer. The result is the same with tokenizer multiprocessing off and on.
I have (absolutely) no idea about the distribution, but since this issue occurs on all of my datasets(regardless of files), I don't think distribution is the problems.

I can use up to 16 cores.
2020-10-26T12:05:45Z
https://github.com/huggingface/datasets/issues/758#issuecomment-716504336
https://api.github.com/repos/huggingface/datasets/issues/758
MDEyOklzc3VlQ29tbWVudDcxNjUwNDMzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/716504336/reactions
2020-10-26T12:05:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/716504336
https://avatars.githubusercontent.com/u/17930170?v=4
https://api.github.com/users/ksjae/events{/privacy}
https://api.github.com/users/ksjae/followers
https://api.github.com/users/ksjae/following{/other_user}
https://api.github.com/users/ksjae/gists{/gist_id}
https://github.com/ksjae
ksjae
MDQ6VXNlcjE3OTMwMTcw
https://api.github.com/users/ksjae/orgs
https://api.github.com/users/ksjae/received_events
https://api.github.com/users/ksjae/repos
https://api.github.com/users/ksjae/starred{/owner}{/repo}
https://api.github.com/users/ksjae/subscriptions
User
https://api.github.com/users/ksjae
public
MEMBER
Ok weird, I don't manage to reproduce this issue on my side.
Does it happen even with `num_proc=2` for example ?
Also could you provide more details about your OS and the versions of tokenizers/datasets/multiprocess that you're using ?
2020-10-26T12:51:06Z
https://github.com/huggingface/datasets/issues/758#issuecomment-716525233
https://api.github.com/repos/huggingface/datasets/issues/758
MDEyOklzc3VlQ29tbWVudDcxNjUyNTIzMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/716525233/reactions
2020-10-26T12:51:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/716525233
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Yes, I can confirm it also happens with ```num_proc=2```.
```
tokenizers               0.9.2
datasets                 1.1.2
multiprocess             0.70.10
```
```
Linux nipa2020-0629 4.4.0-178-generic #208-Ubuntu SMP Sun Apr 5 23:45:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
```
2020-10-26T23:19:46Z
https://github.com/huggingface/datasets/issues/758#issuecomment-716876935
https://api.github.com/repos/huggingface/datasets/issues/758
MDEyOklzc3VlQ29tbWVudDcxNjg3NjkzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/716876935/reactions
2020-10-26T23:19:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/716876935
https://avatars.githubusercontent.com/u/17930170?v=4
https://api.github.com/users/ksjae/events{/privacy}
https://api.github.com/users/ksjae/followers
https://api.github.com/users/ksjae/following{/other_user}
https://api.github.com/users/ksjae/gists{/gist_id}
https://github.com/ksjae
ksjae
MDQ6VXNlcjE3OTMwMTcw
https://api.github.com/users/ksjae/orgs
https://api.github.com/users/ksjae/received_events
https://api.github.com/users/ksjae/repos
https://api.github.com/users/ksjae/starred{/owner}{/repo}
https://api.github.com/users/ksjae/subscriptions
User
https://api.github.com/users/ksjae
public
MEMBER
I can't reproduce on my side unfortunately with the same versions.

Do you have issues when doing multiprocessing with python ?
```python
from tqdm.auto import tqdm
from multiprocess import Pool, RLock

def process_data(shard):
    # implement

num_proc = 8
shards = [] # implement, this must be a list of size num_proc

with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
    results = [pool.apply_async(process_data, shard=shard) for shard in shards]
    transformed_shards = [r.get() for r in results]
```
2020-10-27T15:47:05Z
https://github.com/huggingface/datasets/issues/758#issuecomment-717335889
https://api.github.com/repos/huggingface/datasets/issues/758
MDEyOklzc3VlQ29tbWVudDcxNzMzNTg4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/717335889/reactions
2020-10-27T15:47:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/717335889
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Nah, I'll just wait a few hours. Thank you for helping, though.
2020-10-28T03:59:45Z
https://github.com/huggingface/datasets/issues/758#issuecomment-717680543
https://api.github.com/repos/huggingface/datasets/issues/758
MDEyOklzc3VlQ29tbWVudDcxNzY4MDU0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/717680543/reactions
2020-10-28T03:59:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/717680543
https://avatars.githubusercontent.com/u/17930170?v=4
https://api.github.com/users/ksjae/events{/privacy}
https://api.github.com/users/ksjae/followers
https://api.github.com/users/ksjae/following{/other_user}
https://api.github.com/users/ksjae/gists{/gist_id}
https://github.com/ksjae
ksjae
MDQ6VXNlcjE3OTMwMTcw
https://api.github.com/users/ksjae/orgs
https://api.github.com/users/ksjae/received_events
https://api.github.com/users/ksjae/repos
https://api.github.com/users/ksjae/starred{/owner}{/repo}
https://api.github.com/users/ksjae/subscriptions
User
https://api.github.com/users/ksjae
public
<img width="721" alt="image" src="https://user-images.githubusercontent.com/17930170/97066109-776d0d00-15ed-11eb-8bba-bb4d2e0fcc33.png">
The code I am using is
```

        dataset = load_dataset("text", data_files=[file_path], split='train')
        dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                                truncation=True, max_length=args.block_size), num_proc=8)
        dataset.set_format(type='torch', columns=['input_ids'])
        dataset.save_to_disk(file_path+'.arrow')
```
