num_proc parallelization works only for first ~10s.
MEMBER
Hi, can you check if the processes are still alive ? It's a bit weird because `datasets` does check if processes crash and return an error in that case
2025-04-15T12:47:28Z
https://github.com/huggingface/datasets/issues/7518#issuecomment-2804939077
https://api.github.com/repos/huggingface/datasets/issues/7518
IC_kwDODunzps6nL_lF
https://api.github.com/repos/huggingface/datasets/issues/comments/2804939077/reactions
2025-04-15T12:47:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2804939077
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thank you for reverting quickly. I digged a bit, and realized my disk's IOPS is also limited - which is causing this. will check further and report if it's an issue of hf datasets' side or mine. 
2025-04-15T13:11:55Z
https://github.com/huggingface/datasets/issues/7518#issuecomment-2805012922
https://api.github.com/repos/huggingface/datasets/issues/7518
IC_kwDODunzps6nMRm6
https://api.github.com/repos/huggingface/datasets/issues/comments/2805012922/reactions
2025-04-15T13:12:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2805012922
https://avatars.githubusercontent.com/u/33901783?v=4
https://api.github.com/users/pshishodiaa/events{/privacy}
https://api.github.com/users/pshishodiaa/followers
https://api.github.com/users/pshishodiaa/following{/other_user}
https://api.github.com/users/pshishodiaa/gists{/gist_id}
https://github.com/pshishodiaa
pshishodiaa
MDQ6VXNlcjMzOTAxNzgz
https://api.github.com/users/pshishodiaa/orgs
https://api.github.com/users/pshishodiaa/received_events
https://api.github.com/users/pshishodiaa/repos
https://api.github.com/users/pshishodiaa/starred{/owner}{/repo}
https://api.github.com/users/pshishodiaa/subscriptions
User
https://api.github.com/users/pshishodiaa
public
### Describe the bug

When I try to load an already downloaded dataset with num_proc=64, the speed is very high for the first 10-20 seconds acheiving 30-40K samples / s, and 100% utilization for all cores but it soon drops to <= 1000 with almost 0% utilization for most cores. 

### Steps to reproduce the bug

```
// download dataset with cli
!huggingface-cli download --repo-type dataset timm/imagenet-1k-wds --max-workers 32

from datasets import load_dataset
ds = load_dataset("timm/imagenet-1k-wds", num_proc=64)
```


### Expected behavior

100% core utilization throughout. 

### Environment info

Azure A100-80GB, 16 cores VM

![Image](https://github.com/user-attachments/assets/69d00fe3-d720-4474-9439-21e046d85034)