Refactor `Dataset.map` to reuse cache files mapped with different `num_proc`
NONE
@lhoestq please let me know what you think about this.
2025-03-04T06:33:40Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2696360839
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6gtzOH
https://api.github.com/repos/huggingface/datasets/issues/comments/2696360839/reactions
2025-03-04T06:33:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2696360839
https://avatars.githubusercontent.com/u/27844407?v=4
https://api.github.com/users/ringohoffman/events{/privacy}
https://api.github.com/users/ringohoffman/followers
https://api.github.com/users/ringohoffman/following{/other_user}
https://api.github.com/users/ringohoffman/gists{/gist_id}
https://github.com/ringohoffman
ringohoffman
MDQ6VXNlcjI3ODQ0NDA3
https://api.github.com/users/ringohoffman/orgs
https://api.github.com/users/ringohoffman/received_events
https://api.github.com/users/ringohoffman/repos
https://api.github.com/users/ringohoffman/starred{/owner}{/repo}
https://api.github.com/users/ringohoffman/subscriptions
User
https://api.github.com/users/ringohoffman
public
NONE
It looks like I can't change the merge target to #7435, so it will look like there is a bunch of extra stuff until #7435 is in main.
2025-03-04T22:13:16Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2699066911
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6g4H4f
https://api.github.com/repos/huggingface/datasets/issues/comments/2699066911/reactions
2025-03-04T22:13:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2699066911
https://avatars.githubusercontent.com/u/27844407?v=4
https://api.github.com/users/ringohoffman/events{/privacy}
https://api.github.com/users/ringohoffman/followers
https://api.github.com/users/ringohoffman/following{/other_user}
https://api.github.com/users/ringohoffman/gists{/gist_id}
https://github.com/ringohoffman
ringohoffman
MDQ6VXNlcjI3ODQ0NDA3
https://api.github.com/users/ringohoffman/orgs
https://api.github.com/users/ringohoffman/received_events
https://api.github.com/users/ringohoffman/repos
https://api.github.com/users/ringohoffman/starred{/owner}{/repo}
https://api.github.com/users/ringohoffman/subscriptions
User
https://api.github.com/users/ringohoffman
public
NONE
@lhoestq Thanks so much for reviewing #7435! Now that that's merged, I think this PR is ready!! Can you kick off CI when you get the chance?
2025-03-12T17:29:47Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2718616120
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6iCso4
https://api.github.com/repos/huggingface/datasets/issues/comments/2718616120/reactions
2025-03-12T17:29:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2718616120
https://avatars.githubusercontent.com/u/27844407?v=4
https://api.github.com/users/ringohoffman/events{/privacy}
https://api.github.com/users/ringohoffman/followers
https://api.github.com/users/ringohoffman/following{/other_user}
https://api.github.com/users/ringohoffman/gists{/gist_id}
https://github.com/ringohoffman
ringohoffman
MDQ6VXNlcjI3ODQ0NDA3
https://api.github.com/users/ringohoffman/orgs
https://api.github.com/users/ringohoffman/received_events
https://api.github.com/users/ringohoffman/repos
https://api.github.com/users/ringohoffman/starred{/owner}{/repo}
https://api.github.com/users/ringohoffman/subscriptions
User
https://api.github.com/users/ringohoffman
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7434). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2025-03-12T17:35:01Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2718628584
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6iCvro
https://api.github.com/repos/huggingface/datasets/issues/comments/2718628584/reactions
2025-03-12T17:35:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2718628584
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
Do you mind kicking off CI again?
2025-03-14T12:16:28Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2724511811
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6iZMBD
https://api.github.com/repos/huggingface/datasets/issues/comments/2724511811/reactions
2025-03-14T12:16:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2724511811
https://avatars.githubusercontent.com/u/27844407?v=4
https://api.github.com/users/ringohoffman/events{/privacy}
https://api.github.com/users/ringohoffman/followers
https://api.github.com/users/ringohoffman/following{/other_user}
https://api.github.com/users/ringohoffman/gists{/gist_id}
https://github.com/ringohoffman
ringohoffman
MDQ6VXNlcjI3ODQ0NDA3
https://api.github.com/users/ringohoffman/orgs
https://api.github.com/users/ringohoffman/received_events
https://api.github.com/users/ringohoffman/repos
https://api.github.com/users/ringohoffman/starred{/owner}{/repo}
https://api.github.com/users/ringohoffman/subscriptions
User
https://api.github.com/users/ringohoffman
public
NONE
The change I made to support windows paths in 637c1600fe7dd601eff571fda446937bd96c5c84 ended up breaking causing these tests in [tests/test_data_files.py](https://github.com/huggingface/datasets/actions/runs/13858546629/job/38781008643#step:10:6991). When I removed `glob_pattern_to_regex` in 583c28e7560b9d6db2e13048731f41ec8fa11361, none of the tests failed. So I'm thinking the `unicode_escape` may be handling the what `glob_pattern_to_regex` was doing.

2025-03-14T18:58:54Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2725519004
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6idB6c
https://api.github.com/repos/huggingface/datasets/issues/comments/2725519004/reactions
2025-03-14T18:58:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2725519004
https://avatars.githubusercontent.com/u/27844407?v=4
https://api.github.com/users/ringohoffman/events{/privacy}
https://api.github.com/users/ringohoffman/followers
https://api.github.com/users/ringohoffman/following{/other_user}
https://api.github.com/users/ringohoffman/gists{/gist_id}
https://github.com/ringohoffman
ringohoffman
MDQ6VXNlcjI3ODQ0NDA3
https://api.github.com/users/ringohoffman/orgs
https://api.github.com/users/ringohoffman/received_events
https://api.github.com/users/ringohoffman/repos
https://api.github.com/users/ringohoffman/starred{/owner}{/repo}
https://api.github.com/users/ringohoffman/subscriptions
User
https://api.github.com/users/ringohoffman
public
NONE
@lhoestq will you have a chance to review this today?
2025-03-21T12:37:15Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2743245913
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6jgpxZ
https://api.github.com/repos/huggingface/datasets/issues/comments/2743245913/reactions
2025-03-21T12:37:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2743245913
https://avatars.githubusercontent.com/u/27844407?v=4
https://api.github.com/users/ringohoffman/events{/privacy}
https://api.github.com/users/ringohoffman/followers
https://api.github.com/users/ringohoffman/following{/other_user}
https://api.github.com/users/ringohoffman/gists{/gist_id}
https://github.com/ringohoffman
ringohoffman
MDQ6VXNlcjI3ODQ0NDA3
https://api.github.com/users/ringohoffman/orgs
https://api.github.com/users/ringohoffman/received_events
https://api.github.com/users/ringohoffman/repos
https://api.github.com/users/ringohoffman/starred{/owner}{/repo}
https://api.github.com/users/ringohoffman/subscriptions
User
https://api.github.com/users/ringohoffman
public
NONE
Any update?
2025-05-12T10:05:51Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2871889040
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6rLYyQ
https://api.github.com/repos/huggingface/datasets/issues/comments/2871889040/reactions
2025-05-12T10:05:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2871889040
https://avatars.githubusercontent.com/u/30512160?v=4
https://api.github.com/users/trotsky1997/events{/privacy}
https://api.github.com/users/trotsky1997/followers
https://api.github.com/users/trotsky1997/following{/other_user}
https://api.github.com/users/trotsky1997/gists{/gist_id}
https://github.com/trotsky1997
trotsky1997
MDQ6VXNlcjMwNTEyMTYw
https://api.github.com/users/trotsky1997/orgs
https://api.github.com/users/trotsky1997/received_events
https://api.github.com/users/trotsky1997/repos
https://api.github.com/users/trotsky1997/starred{/owner}{/repo}
https://api.github.com/users/trotsky1997/subscriptions
User
https://api.github.com/users/trotsky1997
public
NONE
> LGTM and sorry for the delay !
> 
> note that the CI failures are unrelated to this PR :)

Great job!
2025-05-12T23:35:56Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2874559460
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6rVkvk
https://api.github.com/repos/huggingface/datasets/issues/comments/2874559460/reactions
2025-05-12T23:35:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2874559460
https://avatars.githubusercontent.com/u/30512160?v=4
https://api.github.com/users/trotsky1997/events{/privacy}
https://api.github.com/users/trotsky1997/followers
https://api.github.com/users/trotsky1997/following{/other_user}
https://api.github.com/users/trotsky1997/gists{/gist_id}
https://github.com/trotsky1997
trotsky1997
MDQ6VXNlcjMwNTEyMTYw
https://api.github.com/users/trotsky1997/orgs
https://api.github.com/users/trotsky1997/received_events
https://api.github.com/users/trotsky1997/repos
https://api.github.com/users/trotsky1997/starred{/owner}{/repo}
https://api.github.com/users/trotsky1997/subscriptions
User
https://api.github.com/users/trotsky1997
public
MEMBER
great job to @ringohoffman ! ;)
2025-05-14T10:45:09Z
https://github.com/huggingface/datasets/pull/7434#issuecomment-2879715029
https://api.github.com/repos/huggingface/datasets/issues/7434
IC_kwDODunzps6rpPbV
https://api.github.com/repos/huggingface/datasets/issues/comments/2879715029/reactions
2025-05-14T10:45:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2879715029
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Fixes #7433

This refactor unifies `num_proc is None or num_proc == 1` and `num_proc > 1`; instead of handling them completely separately where one uses a list of kwargs and shards and the other just uses a single set of kwargs and `self`, by wrapping the `num_proc == 1` case in a list and making the difference just whether or not you use a pool, you set up either case to be able to load each other's cache files just by changing `num_shards`; `num_proc == 1` can sequentially load the shards of a dataset mapped `num_shards > 1` and map any missing shards

Other than the structural refactor, the main contribution of this PR is `existing_cache_file_map`, which uses a regex of `cache_file_name` and `suffix_template` to find existing cache files, grouped by their `num_shards`; using this data structure, we can reset `num_shards` to an existing set of cache files, and load them accordingly