Unpack `dl_manager.iter_files` to allow parallization
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-07-04T13:22:58Z
https://github.com/huggingface/datasets/pull/4625#issuecomment-1173817880
https://api.github.com/repos/huggingface/datasets/issues/4625
IC_kwDODunzps5F9woY
https://api.github.com/repos/huggingface/datasets/issues/comments/1173817880/reactions
2022-07-05T11:11:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1173817880
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Cool thanks ! Yup it sounds like the right solution.

It looks like `_generate_tables` needs to be updated as well to fix the CI
2022-07-04T14:22:45Z
https://github.com/huggingface/datasets/pull/4625#issuecomment-1173879009
https://api.github.com/repos/huggingface/datasets/issues/4625
IC_kwDODunzps5F9_jh
https://api.github.com/repos/huggingface/datasets/issues/comments/1173879009/reactions
2022-07-05T07:30:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1173879009
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Iterate over data files outside `dl_manager.iter_files` to allow parallelization in streaming mode.

(The issue reported [here](https://discuss.huggingface.co/t/dataset-only-have-n-shard-1-when-has-multiple-shards-in-repo/19887))

PS: Another option would be to override `FilesIterable.__getitem__` to make it indexable and check for that type in `_shard_kwargs` and `n_shards,` but IMO this solution adds too much unnecessary complexity.  