.map() cache is wrongfully reused - only happens when the mapping function is imported
MEMBER
Hi ! Thanks for reporting. Indeed this is a current limitation of the usage we have of `dill` in `datasets`. I'd suggest you use your workaround for now until we find a way to fix this. Maybe functions that are not coming from a module not installed with pip should be dumped completely, rather than only taking their locations into account
2021-12-06T14:17:23Z
https://github.com/huggingface/datasets/issues/3297#issuecomment-986817412
https://api.github.com/repos/huggingface/datasets/issues/3297
IC_kwDODunzps460aOE
https://api.github.com/repos/huggingface/datasets/issues/comments/986817412/reactions
2021-12-06T14:17:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/986817412
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I agree. Sounds like a solution for it would be pretty dirty, even [cloudpickle](https://stackoverflow.com/a/16891169) doesn't help in this case.
In the meanwhile I think that adding a warning and the workaround somewhere in the documentation can be helpful.
2021-12-06T23:45:39Z
https://github.com/huggingface/datasets/issues/3297#issuecomment-987377402
https://api.github.com/repos/huggingface/datasets/issues/3297
IC_kwDODunzps462i76
https://api.github.com/repos/huggingface/datasets/issues/comments/987377402/reactions
2021-12-06T23:45:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/987377402
https://avatars.githubusercontent.com/u/13485709?v=4
https://api.github.com/users/eladsegal/events{/privacy}
https://api.github.com/users/eladsegal/followers
https://api.github.com/users/eladsegal/following{/other_user}
https://api.github.com/users/eladsegal/gists{/gist_id}
https://github.com/eladsegal
eladsegal
MDQ6VXNlcjEzNDg1NzA5
https://api.github.com/users/eladsegal/orgs
https://api.github.com/users/eladsegal/received_events
https://api.github.com/users/eladsegal/repos
https://api.github.com/users/eladsegal/starred{/owner}{/repo}
https://api.github.com/users/eladsegal/subscriptions
User
https://api.github.com/users/eladsegal
public
CONTRIBUTOR
For anyone interested, I see that with `dill==0.3.6` the workaround I suggested doesn't work anymore.
I opened an issue about it: https://github.com/uqfoundation/dill/issues/572.

 
2023-01-30T12:40:17Z
https://github.com/huggingface/datasets/issues/3297#issuecomment-1408546750
https://api.github.com/repos/huggingface/datasets/issues/3297
IC_kwDODunzps5T9Le-
https://api.github.com/repos/huggingface/datasets/issues/comments/1408546750/reactions
2023-01-30T12:40:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1408546750
https://avatars.githubusercontent.com/u/13485709?v=4
https://api.github.com/users/eladsegal/events{/privacy}
https://api.github.com/users/eladsegal/followers
https://api.github.com/users/eladsegal/following{/other_user}
https://api.github.com/users/eladsegal/gists{/gist_id}
https://github.com/eladsegal
eladsegal
MDQ6VXNlcjEzNDg1NzA5
https://api.github.com/users/eladsegal/orgs
https://api.github.com/users/eladsegal/received_events
https://api.github.com/users/eladsegal/repos
https://api.github.com/users/eladsegal/starred{/owner}{/repo}
https://api.github.com/users/eladsegal/subscriptions
User
https://api.github.com/users/eladsegal
public
NONE
Is there a plan for this issue or some progress made on this issue?

Having tons of transformations on a dataset doesn't allow to place it in the same file as loading it..

Also, currently there is no workaround available
2025-07-30T21:01:38Z
https://github.com/huggingface/datasets/issues/3297#issuecomment-3137808220
https://api.github.com/repos/huggingface/datasets/issues/3297
IC_kwDODunzps67Bydc
https://api.github.com/repos/huggingface/datasets/issues/comments/3137808220/reactions
2025-07-30T21:01:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3137808220
https://avatars.githubusercontent.com/u/207981996?v=4
https://api.github.com/users/amitabe-amazon/events{/privacy}
https://api.github.com/users/amitabe-amazon/followers
https://api.github.com/users/amitabe-amazon/following{/other_user}
https://api.github.com/users/amitabe-amazon/gists{/gist_id}
https://github.com/amitabe-amazon
amitabe-amazon
U_kgDODGWNrA
https://api.github.com/users/amitabe-amazon/orgs
https://api.github.com/users/amitabe-amazon/received_events
https://api.github.com/users/amitabe-amazon/repos
https://api.github.com/users/amitabe-amazon/starred{/owner}{/repo}
https://api.github.com/users/amitabe-amazon/subscriptions
User
https://api.github.com/users/amitabe-amazon
public
MEMBER
Hi ! a workaround is to define the `fingerprint` uses to locate the cache file manually:

```python
ds = ds.map(..., new_fingerprint=new_fingerprint)
```

**PS: make sure to update `new_fingerprint` every time you change your map() function or it will reload previous results from the cache**
2025-07-31T16:29:29Z
https://github.com/huggingface/datasets/issues/3297#issuecomment-3140594331
https://api.github.com/repos/huggingface/datasets/issues/3297
IC_kwDODunzps67Maqb
https://api.github.com/repos/huggingface/datasets/issues/comments/3140594331/reactions
2025-07-31T16:29:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3140594331
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug
When `.map` is used with a mapping function that is imported, the cache is reused even if the mapping function has been modified.
The reason for this is that `dill` that is used for creating the fingerprint [pickles imported functions by reference](https://stackoverflow.com/a/67851411).

I guess it is not a widespread case, but it can still lead to unwanted results unnoticeably. 

## Steps to reproduce the bug
Create files `a.py` and `b.py`:
```python
# a.py
from datasets import load_dataset

def main():
    squad = load_dataset("squad")
    squad.map(mapping_func, batched=True)

def mapping_func(examples):
    ID_LENGTH = 4
    examples["id"] = [id_[:ID_LENGTH] for id_ in examples["id"]]
    return examples

if __name__ == "__main__":
    main()
```
```python
# b.py
from datasets import load_dataset
from a import mapping_func

def main():
    squad = load_dataset("squad")
    squad.map(mapping_func, batched=True)

if __name__ == "__main__":
    main()
```
Run `python b.py` twice: In the first run you will see tqdm bars showing that the data is processed, and in the second run you will see "Loading cached processed dataset at...".
Now change `ID_LENGTH` to another number in order to change the mapping function, and run `python b.py` again. You'll see that `.map` loads from the cache the result of the previous mapping function.

## Expected results
Run `python a.py` twice: In the first run you will see tqdm bars showing that the data is processed, and in the second run you will see "Loading cached processed dataset at...".
Now change `ID_LENGTH` to another number in order to change the mapping function, and run `python a.py` again. You'll see that the dataset is being processed and that there's no reuse of the previous mapping function result.

## Workaround
Put the mapping function inside a dummy class as a static method:
```python
# a.py
class MappingFuncClass:
    @staticmethod
    def mapping_func(examples):
        ID_LENGTH = 4
        examples["id"] = [id_[:ID_LENGTH] for id_ in examples["id"]]
        return examples
```
```python
# b.py
from datasets import load_dataset
from a import MappingFuncClass

def main():
    squad = load_dataset("squad")
    squad.map(MappingFuncClass.mapping_func, batched=True)

if __name__ == "__main__":
    main()
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1
