Running `datase.map` with `num_proc > 1` uses a lot of memory
MEMBER
Thanks for reporting ! We are working on this and we'll do a patch release very soon.
2021-04-26T08:23:15Z
https://github.com/huggingface/datasets/issues/2256#issuecomment-826621470
https://api.github.com/repos/huggingface/datasets/issues/2256
MDEyOklzc3VlQ29tbWVudDgyNjYyMTQ3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/826621470/reactions
2021-04-26T08:23:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826621470
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
We did a patch release to fix this issue.
It should be fixed in the new version 1.6.1

Thanks again for reporting and for the details :)
2021-04-26T17:12:02Z
https://github.com/huggingface/datasets/issues/2256#issuecomment-827006521
https://api.github.com/repos/huggingface/datasets/issues/2256
MDEyOklzc3VlQ29tbWVudDgyNzAwNjUyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/827006521/reactions
2021-04-26T17:12:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/827006521
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug
Running `datase.map` with `num_proc > 1`  leads to a tremendous memory usage that requires swapping on disk and it becomes very slow.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dstc8_datset = load_dataset("roskoN/dstc8-reddit-corpus", keep_in_memory=False)


def _prepare_sample(batch):
    return {"input_ids": list(), "attention_mask": list()}


for split_name, dataset_split in list(dstc8_datset.items()):
    print(f"Processing {split_name}")
    encoded_dataset_split = dataset_split.map(
        function=_prepare_sample,
        batched=True,
        num_proc=4,
        remove_columns=dataset_split.column_names,
        batch_size=10,
        writer_batch_size=10,
        keep_in_memory=False,
    )
    print(encoded_dataset_split)

    path = f"./data/encoded_{split_name}"

    encoded_dataset_split.save_to_disk(path)
```

## Expected results
Memory usage should stay within reasonable boundaries.


## Actual results
This is htop-output from running the provided script.

![image](https://user-images.githubusercontent.com/8143425/115954836-66954980-a4f3-11eb-8340-0153bdc3a475.png)

## Versions
```
- Datasets: 1.6.0
- Python: 3.8.8 (default, Apr 13 2021, 19:58:26)
[GCC 7.3.0]
- Platform: Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.10
```
Running on WSL2
