More control over TQDM when using map/filter with multiple processes
COLLABORATOR
Hi,

It's hard to provide an API that would cover all use-cases with tqdm in this project.

However, you can make it work by defining a custom decorator (a bit hacky tho) as follows:
```python
import datasets

def progress_only_on_rank_0(func):
    def wrapper(*args, **kwargs):
        rank = kwargs.get("rank")
        disable_tqdm = kwargs.get("disable_tqdm", False)
        disable_tqdm = True if rank is not None and rank > 0 else disable_tqdm
        kwargs["disable_tqdm"] = disable_tqdm
        return func(*args, **kwargs)
    return wrapper
    
datasets.Dataset._map_single = progress_only_on_rank_0(datasets.Dataset._map_single)
``` 

EDIT: Ups, closed by accident.

Thanks for the provided links. `Trainer` requires this for training in multi-node distributed setting. However, `Dataset.map` doesn't support that yet.

Do you have an API for this in mind? `Dataset.map` is already bloated with the arguments, so IMO it's not a good idea to add a new arg there.


2021-11-02T10:32:07Z
https://github.com/huggingface/datasets/issues/3177#issuecomment-957313551
https://api.github.com/repos/huggingface/datasets/issues/3177
IC_kwDODunzps45D3IP
https://api.github.com/repos/huggingface/datasets/issues/comments/957313551/reactions
2021-11-02T11:07:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/957313551
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Inspiration may be found at `transformers`.

https://github.com/huggingface/transformers/blob/4a394cf53f05e73ab9bbb4b179a40236a5ffe45a/src/transformers/trainer.py#L1231-L1233

To get unique IDs for each worker, see https://stackoverflow.com/a/10192611/1150683
2021-11-02T10:45:23Z
https://github.com/huggingface/datasets/issues/3177#issuecomment-957323182
https://api.github.com/repos/huggingface/datasets/issues/3177
IC_kwDODunzps45D5eu
https://api.github.com/repos/huggingface/datasets/issues/comments/957323182/reactions
2021-11-02T11:38:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/957323182
https://avatars.githubusercontent.com/u/2779410?v=4
https://api.github.com/users/BramVanroy/events{/privacy}
https://api.github.com/users/BramVanroy/followers
https://api.github.com/users/BramVanroy/following{/other_user}
https://api.github.com/users/BramVanroy/gists{/gist_id}
https://github.com/BramVanroy
BramVanroy
MDQ6VXNlcjI3Nzk0MTA=
https://api.github.com/users/BramVanroy/orgs
https://api.github.com/users/BramVanroy/received_events
https://api.github.com/users/BramVanroy/repos
https://api.github.com/users/BramVanroy/starred{/owner}{/repo}
https://api.github.com/users/BramVanroy/subscriptions
User
https://api.github.com/users/BramVanroy
public
It would help with the clutter in my terminal if tqdm is only shown for rank 0 when using `num_proces>0` in the map and filter methods of datasets.

```python
dataset.map(lambda examples: tokenize(examples["text"]), batched=True, num_proc=6)
```

The above snippet leads to a lot of TQDM bars and depending on your terminal, these will not overwrite but keep pushing each other down.

```
 #0:   0%|          | 0/13 [00:00<?, ?ba/s]
 #1:   0%|          | 0/13 [00:00<?, ?ba/s]
 #2:   0%|          | 0/13 [00:00<?, ?ba/s]
 #3:   0%|          | 0/13 [00:00<?, ?ba/s]
 #4:   0%|          | 0/13 [00:00<?, ?ba/s]
 #5:   0%|          | 0/13 [00:00<?, ?ba/s]
 #0:   8%|          | 1/13 [00:00<?, ?ba/s]
 #1:   8%|          | 1/13 [00:00<?, ?ba/s]
...
```

Instead, it would be welcome if we had the option to only show the progress of rank 0.