Support of num_workers (multiprocessing) in map for IterableDataset
CONTRIBUTOR
I was curious about the same - since map is applied on the fly I was assuming that setting num_workers>1 in DataLoader would effectively do the map in parallel, have you tried that?
2024-10-03T09:54:13Z
https://github.com/huggingface/datasets/issues/7193#issuecomment-2390995726
https://api.github.com/repos/huggingface/datasets/issues/7193
IC_kwDODunzps6Og7MO
https://api.github.com/repos/huggingface/datasets/issues/comments/2390995726/reactions
2024-10-03T09:54:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2390995726
https://avatars.githubusercontent.com/u/5719745?v=4
https://api.github.com/users/alex-hh/events{/privacy}
https://api.github.com/users/alex-hh/followers
https://api.github.com/users/alex-hh/following{/other_user}
https://api.github.com/users/alex-hh/gists{/gist_id}
https://github.com/alex-hh
alex-hh
MDQ6VXNlcjU3MTk3NDU=
https://api.github.com/users/alex-hh/orgs
https://api.github.com/users/alex-hh/received_events
https://api.github.com/users/alex-hh/repos
https://api.github.com/users/alex-hh/starred{/owner}{/repo}
https://api.github.com/users/alex-hh/subscriptions
User
https://api.github.com/users/alex-hh
public
### Feature request

Currently, IterableDataset doesn't support setting num_worker in .map(), which results in slow processing here. Could we add support for it? As .map() can be run in the batch fashion (e.g., batch_size is default to 1000 in datasets), it seems to be doable for IterableDataset as the regular Dataset.

### Motivation

Improving data processing efficiency

### Your contribution

Testing