Use same `num_proc` for dataset download and generation
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-11-25T15:42:59Z
https://github.com/huggingface/datasets/pull/5300#issuecomment-1327641016
https://api.github.com/repos/huggingface/datasets/issues/5300
IC_kwDODunzps5PIjG4
https://api.github.com/repos/huggingface/datasets/issues/comments/1327641016/reactions
2022-12-07T12:55:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1327641016
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
I noticed this bug the other day and was going to look into it! "Where are these processes coming from?" ;-)
2022-11-25T15:52:04Z
https://github.com/huggingface/datasets/pull/5300#issuecomment-1327647813
https://api.github.com/repos/huggingface/datasets/issues/5300
IC_kwDODunzps5PIkxF
https://api.github.com/repos/huggingface/datasets/issues/comments/1327647813/reactions
2022-11-25T15:52:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1327647813
https://avatars.githubusercontent.com/u/445208?v=4
https://api.github.com/users/alexjc/events{/privacy}
https://api.github.com/users/alexjc/followers
https://api.github.com/users/alexjc/following{/other_user}
https://api.github.com/users/alexjc/gists{/gist_id}
https://github.com/alexjc
alexjc
MDQ6VXNlcjQ0NTIwOA==
https://api.github.com/users/alexjc/orgs
https://api.github.com/users/alexjc/received_events
https://api.github.com/users/alexjc/repos
https://api.github.com/users/alexjc/starred{/owner}{/repo}
https://api.github.com/users/alexjc/subscriptions
User
https://api.github.com/users/alexjc
public
Use the same `num_proc` value for data download and generation. Additionally, do not set `num_proc` to 16 in `DownloadManager` by default (`num_proc` now has to be specified explicitly).