`map()` stops every 1000 steps
MEMBER
Hi !

> It starts using all the cores (I am not sure why because I did not pass num_proc)

The tokenizer uses Rust code that is multithreaded. And maybe the `feature_extractor` might run some things in parallel as well - but I'm not super familiar with its internals.

> then progress bar stops at every 1k steps. (starts using a single core)

Every 1000 examples we flush the processed examples to disk. It is this way because Arrow is a columnar format: you must write data chunk by chunk. The processing in on hold while writing right now - maybe this can be improved in the future.
2022-12-07T22:08:13Z
https://github.com/huggingface/datasets/issues/5338#issuecomment-1341656380
https://api.github.com/repos/huggingface/datasets/issues/5338
IC_kwDODunzps5P-A08
https://api.github.com/repos/huggingface/datasets/issues/comments/1341656380/reactions
2022-12-07T22:09:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1341656380
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq 
Thanks for the explanation! it was so helpful! Let me check why `feature_extractor` is running on multiple cpus.
2022-12-10T00:39:28Z
https://github.com/huggingface/datasets/issues/5338#issuecomment-1344947801
https://api.github.com/repos/huggingface/datasets/issues/5338
IC_kwDODunzps5QKkZZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1344947801/reactions
2022-12-10T00:39:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1344947801
https://avatars.githubusercontent.com/u/43239645?v=4
https://api.github.com/users/bayartsogt-ya/events{/privacy}
https://api.github.com/users/bayartsogt-ya/followers
https://api.github.com/users/bayartsogt-ya/following{/other_user}
https://api.github.com/users/bayartsogt-ya/gists{/gist_id}
https://github.com/bayartsogt-ya
bayartsogt-ya
MDQ6VXNlcjQzMjM5NjQ1
https://api.github.com/users/bayartsogt-ya/orgs
https://api.github.com/users/bayartsogt-ya/received_events
https://api.github.com/users/bayartsogt-ya/repos
https://api.github.com/users/bayartsogt-ya/starred{/owner}{/repo}
https://api.github.com/users/bayartsogt-ya/subscriptions
User
https://api.github.com/users/bayartsogt-ya
public
NONE
Hey @lhoestq, any news about this flush operation ? This really really slow down most of my .map and .filter operation. I'm sometime taking 10x more time waiting for the flush than actually processing the 1k items. My current workarround is to set `writer_batch_size=len(dataset)`.
2025-02-14T18:10:06Z
https://github.com/huggingface/datasets/issues/5338#issuecomment-2659971676
https://api.github.com/repos/huggingface/datasets/issues/5338
IC_kwDODunzps6ei_Jc
https://api.github.com/repos/huggingface/datasets/issues/comments/2659971676/reactions
2025-02-14T18:10:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2659971676
https://avatars.githubusercontent.com/u/11278197?v=4
https://api.github.com/users/julien-blanchon/events{/privacy}
https://api.github.com/users/julien-blanchon/followers
https://api.github.com/users/julien-blanchon/following{/other_user}
https://api.github.com/users/julien-blanchon/gists{/gist_id}
https://github.com/julien-blanchon
julien-blanchon
MDQ6VXNlcjExMjc4MTk3
https://api.github.com/users/julien-blanchon/orgs
https://api.github.com/users/julien-blanchon/received_events
https://api.github.com/users/julien-blanchon/repos
https://api.github.com/users/julien-blanchon/starred{/owner}{/repo}
https://api.github.com/users/julien-blanchon/subscriptions
User
https://api.github.com/users/julien-blanchon
public
### Describe the bug

I am passing the following `prepare_dataset` function to `Dataset.map` (code is inspired from [here](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/run_speech_recognition_seq2seq_streaming.py#L454))
```python3
def prepare_dataset(batch):
    # load and resample audio data from 48 to 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array 
    batch["input_features"] = feature_extractor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]

    # encode target text to label ids 
    batch["labels"] = tokenizer(batch[text_column]).input_ids
    return batch
...
train_ds = train_ds.map(prepare_dataset)
```

Here is the exact code I am running https://github.com/bayartsogt-ya/whisper-multiple-hf-datasets/blob/main/train.py#L70-L71

It starts using all the cores (I am not sure why because I did not pass `num_proc`)
then progress bar stops at every 1k steps. (starts using a single core)
then come back to using all the cores again.

link to [screen record](https://youtu.be/jPQpQQGp6Gc)

Can someone explain this process and maybe provide a way to improve this pipeline? cc: @lhoestq 

### Steps to reproduce the bug

1. load the dataset
2. create a Whisper processor
3. create a `prepare_dataset` function
4. pass the function to `dataset.map(prepare_dataset)`

### Expected behavior

- Use a single core per a function
- not to stop at some point?

### Environment info

- `datasets` version: 2.7.1.dev0
- Platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.27
- Python version: 3.8.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.2