Map is slow and processes batches one after another
MEMBER
Hi @villmow, thanks for reporting.

Could you please try with the Datasets version 1.6? We released it yesterday and it fixes some issues about the processing speed. You can see the fix implemented by @lhoestq here: #2122.

Once you update Datasets, please confirm if the problem persists.
2021-04-21T16:18:31Z
https://github.com/huggingface/datasets/issues/2243#issuecomment-824188841
https://api.github.com/repos/huggingface/datasets/issues/2243
MDEyOklzc3VlQ29tbWVudDgyNDE4ODg0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/824188841/reactions
2021-04-21T16:18:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/824188841
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Hi @albertvillanova, thanks for the reply. I just tried the new version and the problem still persists. 

Do I need to rebuild the saved dataset (which I load from disk) with the 1.6.0 version of datasets? My script loads this dataset and creates new datasets from it. I tried it without rebuilding.

See this short video of what happens. It does not create all processes at the same time:

https://user-images.githubusercontent.com/2743060/115720139-0da3a500-a37d-11eb-833a-9bbacc70868d.mp4


2021-04-22T13:14:30Z
https://github.com/huggingface/datasets/issues/2243#issuecomment-824831085
https://api.github.com/repos/huggingface/datasets/issues/2243
MDEyOklzc3VlQ29tbWVudDgyNDgzMTA4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/824831085/reactions
2021-04-22T13:14:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/824831085
https://avatars.githubusercontent.com/u/2743060?v=4
https://api.github.com/users/villmow/events{/privacy}
https://api.github.com/users/villmow/followers
https://api.github.com/users/villmow/following{/other_user}
https://api.github.com/users/villmow/gists{/gist_id}
https://github.com/villmow
villmow
MDQ6VXNlcjI3NDMwNjA=
https://api.github.com/users/villmow/orgs
https://api.github.com/users/villmow/received_events
https://api.github.com/users/villmow/repos
https://api.github.com/users/villmow/starred{/owner}{/repo}
https://api.github.com/users/villmow/subscriptions
User
https://api.github.com/users/villmow
public
MEMBER
There can be a bit of delay between the creations of the processes but this delay should be the same for both your `map` calls. We should look into this.
Also if you hav some code that reproduces this issue on google colab that'd be really useful !

Regarding the speed differences:
This looks like a similar issue as https://github.com/huggingface/datasets/issues/1992 who is experiencing the same speed differences between processes.
This is a known bug that we are investigating. As of now I've never managed to reproduce it on my machine so it's pretty hard for me to find where this issue comes from.

2021-04-26T10:03:17Z
https://github.com/huggingface/datasets/issues/2243#issuecomment-826699484
https://api.github.com/repos/huggingface/datasets/issues/2243
MDEyOklzc3VlQ29tbWVudDgyNjY5OTQ4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/826699484/reactions
2021-04-26T10:03:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826699484
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Upgrade to 1.6.1 solved my problem somehow. I did not change any of my code, but now it starts all processes around the same time.
2021-04-30T07:47:06Z
https://github.com/huggingface/datasets/issues/2243#issuecomment-829910759
https://api.github.com/repos/huggingface/datasets/issues/2243
MDEyOklzc3VlQ29tbWVudDgyOTkxMDc1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/829910759/reactions
2021-04-30T07:47:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/829910759
https://avatars.githubusercontent.com/u/2743060?v=4
https://api.github.com/users/villmow/events{/privacy}
https://api.github.com/users/villmow/followers
https://api.github.com/users/villmow/following{/other_user}
https://api.github.com/users/villmow/gists{/gist_id}
https://github.com/villmow
villmow
MDQ6VXNlcjI3NDMwNjA=
https://api.github.com/users/villmow/orgs
https://api.github.com/users/villmow/received_events
https://api.github.com/users/villmow/repos
https://api.github.com/users/villmow/starred{/owner}{/repo}
https://api.github.com/users/villmow/subscriptions
User
https://api.github.com/users/villmow
public
MEMBER
Nice ! I'm glad this works now.
Closing for now, but feel free to re-open if you experience this issue again.
2021-05-03T17:54:32Z
https://github.com/huggingface/datasets/issues/2243#issuecomment-831425795
https://api.github.com/repos/huggingface/datasets/issues/2243
MDEyOklzc3VlQ29tbWVudDgzMTQyNTc5NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/831425795/reactions
2021-05-03T17:54:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/831425795
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug

I have a somewhat unclear bug to me, where I can't figure out what the problem is. The code works as expected on a small subset of my dataset (2000 samples) on my local machine, but when I execute the same code with a larger dataset (1.4 million samples) this problem occurs. Thats why I can't give exact steps to reproduce, I'm sorry. 

I process a large dataset in a two step process. I first call map on a dataset I load from disk and create a new dataset from it. This works like expected and `map` uses all workers I started it with. Then I process the dataset created by the first step, again with `map`, which is really slow and starting only one or two process at a time. Number of processes is the same for both steps.

pseudo code:
```python
ds = datasets.load_from_disk("path")
new_dataset = ds.map(work, batched=True, ...)  # fast uses all processes
final_dataset = new_dataset.map(work2, batched=True, ...)  # slow starts one process after another
```

## Expected results
Second stage should be as fast as the first stage.

## Versions
Paste the output of the following code:
- Datasets: 1.5.0
- Python: 3.8.8 (default, Feb 24 2021, 21:46:12)
- Platform: Linux-5.4.0-60-generic-x86_64-with-glibc2.10    

Do you guys have any idea? Thanks a lot!