Simplify skipping
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-12-17T17:28:07Z
https://github.com/huggingface/datasets/pull/5373#issuecomment-1356345679
https://api.github.com/repos/huggingface/datasets/issues/5373
IC_kwDODunzps5Q2DFP
https://api.github.com/repos/huggingface/datasets/issues/comments/1356345679/reactions
2022-12-18T21:43:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1356345679
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
Was hoping to find a way to speed up the skipping as I'm running into bottlenecks skipping 100M examples on C4 (it takes 12 hours to skip), but didn't find anything better than this small change :(

Maybe there's a way to directly skip whole shards to speed it up? 🧐