Support multi-worker with streaming dataset (IterableDataset).
MEMBER
Hi ! This is a great idea :)
I think we could have something similar to what we have in `datasets.Dataset.map`, i.e. a `num_proc` parameter that tells how many processes to spawn to parallelize the data processing. 

Regarding AUTOTUNE, this could be a nice feature as well, we could see how to add it in a second step
2021-07-15T09:37:34Z
https://github.com/huggingface/datasets/issues/2642#issuecomment-880549605
https://api.github.com/repos/huggingface/datasets/issues/2642
MDEyOklzc3VlQ29tbWVudDg4MDU0OTYwNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/880549605/reactions
2021-07-15T09:37:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/880549605
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Any update on this feature request?
2024-05-03T07:51:23Z
https://github.com/huggingface/datasets/issues/2642#issuecomment-2092489456
https://api.github.com/repos/huggingface/datasets/issues/2642
IC_kwDODunzps58uNrw
https://api.github.com/repos/huggingface/datasets/issues/comments/2092489456/reactions
2024-05-03T07:51:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2092489456
https://avatars.githubusercontent.com/u/4197249?v=4
https://api.github.com/users/memray/events{/privacy}
https://api.github.com/users/memray/followers
https://api.github.com/users/memray/following{/other_user}
https://api.github.com/users/memray/gists{/gist_id}
https://github.com/memray
memray
MDQ6VXNlcjQxOTcyNDk=
https://api.github.com/users/memray/orgs
https://api.github.com/users/memray/received_events
https://api.github.com/users/memray/repos
https://api.github.com/users/memray/starred{/owner}{/repo}
https://api.github.com/users/memray/subscriptions
User
https://api.github.com/users/memray
public
MEMBER
Not yet, I'm happy to provide some guidance if someone wants to give it a shot though.

The code that applies the `map` function is in `iterable_dataset.py`, in `MappedExamplesIterable.__iter__`
2024-05-03T10:11:03Z
https://github.com/huggingface/datasets/issues/2642#issuecomment-2092705436
https://api.github.com/repos/huggingface/datasets/issues/2642
IC_kwDODunzps58vCac
https://api.github.com/repos/huggingface/datasets/issues/comments/2092705436/reactions
2024-05-03T10:11:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2092705436
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
**Is your feature request related to a problem? Please describe.**
The current `.map` does not support multi-process, CPU can become bottleneck if the pre-processing is complex (e.g. t5 span masking).

**Describe the solution you'd like**
Ideally `.map` should support multi-worker like tfds, with `AUTOTUNE`.

**Describe alternatives you've considered**
A simpler solution is to shard the dataset and process it in parallel with pytorch dataloader. The shard does not need to be of equal size.
* https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

**Additional context**
