Caching map result of DatasetDict.
NONE
Has to do with sharding as far as I am aware, num_proc splits the dataset in shards and each process picks up a shard.
2025-07-28T10:54:12Z
https://github.com/huggingface/datasets/issues/6924#issuecomment-3126670068
https://api.github.com/repos/huggingface/datasets/issues/6924
IC_kwDODunzps66XTL0
https://api.github.com/repos/huggingface/datasets/issues/comments/3126670068/reactions
2025-07-28T10:54:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3126670068
https://avatars.githubusercontent.com/u/46669905?v=4
https://api.github.com/users/dimidagd/events{/privacy}
https://api.github.com/users/dimidagd/followers
https://api.github.com/users/dimidagd/following{/other_user}
https://api.github.com/users/dimidagd/gists{/gist_id}
https://github.com/dimidagd
dimidagd
MDQ6VXNlcjQ2NjY5OTA1
https://api.github.com/users/dimidagd/orgs
https://api.github.com/users/dimidagd/received_events
https://api.github.com/users/dimidagd/repos
https://api.github.com/users/dimidagd/starred{/owner}{/repo}
https://api.github.com/users/dimidagd/subscriptions
User
https://api.github.com/users/dimidagd
public
NONE
@dimidagd why wouldn't you redistribute the processes (e.g. param_num_process to sequentially fill fingerprint_num_process) to make things work out? 

Perhaps I'm missing something important.
2025-07-28T11:08:02Z
https://github.com/huggingface/datasets/issues/6924#issuecomment-3126719630
https://api.github.com/repos/huggingface/datasets/issues/6924
IC_kwDODunzps66XfSO
https://api.github.com/repos/huggingface/datasets/issues/comments/3126719630/reactions
2025-07-28T11:08:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3126719630
https://avatars.githubusercontent.com/u/56939432?v=4
https://api.github.com/users/MostHumble/events{/privacy}
https://api.github.com/users/MostHumble/followers
https://api.github.com/users/MostHumble/following{/other_user}
https://api.github.com/users/MostHumble/gists{/gist_id}
https://github.com/MostHumble
MostHumble
MDQ6VXNlcjU2OTM5NDMy
https://api.github.com/users/MostHumble/orgs
https://api.github.com/users/MostHumble/received_events
https://api.github.com/users/MostHumble/repos
https://api.github.com/users/MostHumble/starred{/owner}{/repo}
https://api.github.com/users/MostHumble/subscriptions
User
https://api.github.com/users/MostHumble
public
NONE
I assume data has to be split between the processes somehow, and the decision to split it to num_processes shards is very convenient :). As I understand, the idea behind individual shard caching is being able to recover part of the processing if one of the processes fails. It would make sense to do a hierarchical cache checking though, where irregardless of the amount of processes, there is a check on whether the entire .map result has been cached or not (basically num_proc=1). Downside is more disk space used to cache both the shards and the entire dataset. 
2025-07-28T12:57:33Z
https://github.com/huggingface/datasets/issues/6924#issuecomment-3127123822
https://api.github.com/repos/huggingface/datasets/issues/6924
IC_kwDODunzps66ZB9u
https://api.github.com/repos/huggingface/datasets/issues/comments/3127123822/reactions
2025-07-28T12:57:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3127123822
https://avatars.githubusercontent.com/u/46669905?v=4
https://api.github.com/users/dimidagd/events{/privacy}
https://api.github.com/users/dimidagd/followers
https://api.github.com/users/dimidagd/following{/other_user}
https://api.github.com/users/dimidagd/gists{/gist_id}
https://github.com/dimidagd
dimidagd
MDQ6VXNlcjQ2NjY5OTA1
https://api.github.com/users/dimidagd/orgs
https://api.github.com/users/dimidagd/received_events
https://api.github.com/users/dimidagd/repos
https://api.github.com/users/dimidagd/starred{/owner}{/repo}
https://api.github.com/users/dimidagd/subscriptions
User
https://api.github.com/users/dimidagd
public
Hi!

I'm currenty using the map function to tokenize a somewhat large dataset, so I need to use the cache to save ~25 mins.

Changing num_proc incduces the recomputation of the map, I'm not sure why and if this is excepted behavior? 

here it says, that cached files are loaded sequentially:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3005-L3006

it seems like I can pass in a fingerprint, and load it directly:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3108-L3125

**Environment Setup:**

- Python 3.11.9
- datasets  2.19.1 conda-forge
- Linux 6.1.83-1.el9.elrepo.x86_64

**MRE**
```python
fixed raw_datasets
fixed tokenize_function

tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=9,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )


tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=5,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )
```