Slow #0 when using map to tokenize.
MEMBER
Hi ! Have you tried other values for `preprocessing_num_workers` ? Is it always process 0 that is slower ?
There are no difference between process 0 and the others except that it processes the first shard of the dataset.
2021-04-30T12:33:15Z
https://github.com/huggingface/datasets/issues/2294#issuecomment-830062339
https://api.github.com/repos/huggingface/datasets/issues/2294
MDEyOklzc3VlQ29tbWVudDgzMDA2MjMzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/830062339/reactions
2021-04-30T12:33:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/830062339
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, I have found the reason of it. Before using the map function to tokenize the data, I concatenate the wikipedia and bookcorpus first, like this:
```if args.dataset_name1 is not None:
        dataset1 = load_dataset(args.dataset_name1, args.dataset_config_name1, split="train")
        dataset1 = dataset1.remove_columns('title')
        if args.dataset_name2 is not None:
            dataset2 = load_dataset(args.dataset_name2, args.dataset_config_name2,split="train")
            assert dataset1.features.type == dataset2.features.type, str(dataset1.features.type)+';'+str(dataset2.features.type)
           datasets12 = concatenate_datasets([dataset1, dataset2], split='train')
```
When I just use one datasets, e.g. wikipedia, the problem seems no longer exist:
![image](https://user-images.githubusercontent.com/31714566/116967059-13d24380-ace4-11eb-8d14-b7b9c9a275cc.png)

Bookcorpus has more row numbers than Wikipedia, however, it takes much more time to process each batch of wiki than that of bookcorpus. When we first concatenate two datasets and then use _map_ to process the concatenated datasets,  e.g. `num_proc=5`, process 0 has to process all of the wikipedia data, causing the problem that #0 takes a longer time to finish the job. 

The problem is caused by the different characteristic of different datasets. One solution might be using _map_ first to process two datasets seperately, then concatenate the tokenized and processed datasets before input to the `Dataloader`.


2021-05-04T06:38:06Z
https://github.com/huggingface/datasets/issues/2294#issuecomment-831715244
https://api.github.com/repos/huggingface/datasets/issues/2294
MDEyOklzc3VlQ29tbWVudDgzMTcxNTI0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/831715244/reactions
2021-05-04T06:38:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/831715244
https://avatars.githubusercontent.com/u/31714566?v=4
https://api.github.com/users/VerdureChen/events{/privacy}
https://api.github.com/users/VerdureChen/followers
https://api.github.com/users/VerdureChen/following{/other_user}
https://api.github.com/users/VerdureChen/gists{/gist_id}
https://github.com/VerdureChen
VerdureChen
MDQ6VXNlcjMxNzE0NTY2
https://api.github.com/users/VerdureChen/orgs
https://api.github.com/users/VerdureChen/received_events
https://api.github.com/users/VerdureChen/repos
https://api.github.com/users/VerdureChen/starred{/owner}{/repo}
https://api.github.com/users/VerdureChen/subscriptions
User
https://api.github.com/users/VerdureChen
public
MEMBER
That makes sense ! You can indeed use `map` on both datasets separately and then concatenate.
Another option is to concatenate, then shuffle, and then `map`.
2021-05-04T11:00:11Z
https://github.com/huggingface/datasets/issues/2294#issuecomment-831855639
https://api.github.com/repos/huggingface/datasets/issues/2294
MDEyOklzc3VlQ29tbWVudDgzMTg1NTYzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/831855639/reactions
2021-05-04T11:00:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/831855639
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hi, _datasets_ is really amazing! I am following [run_mlm_no_trainer.py](url) to pre-train BERT, and it uses `tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not args.overwrite_cache,
        )` to tokenize by multiprocessing. However, I have found that when `num_proc`>1，the process _#0_ is much slower than others.
It looks like this:
![image](https://user-images.githubusercontent.com/31714566/116665555-81246280-a9cc-11eb-8a37-6e608ab310d0.png)
It takes more than 12 hours for #0, while others just about half an hour. Could anyone tell me it is normal or not, and is there any methods to speed up it?
