Map cache does not detect function changes in another module
COLLABORATOR
This issue is a duplicate of https://github.com/huggingface/datasets/issues/3297.  This is a limitation of `dill`, a package we use for caching (non-`__main__` module objects are serialized by reference). You can find more info about it here: https://github.com/uqfoundation/dill/issues/424.

In your case, moving 
```
data = datasets.load_dataset('json', data_files=['/tmp/test.json'], split='train')
data = data.map(transform)
``` 
to `test.py`  and setting `transform.__module__ = None` at the end of `dataset.py` should fix the issue.
2023-08-29T14:15:34Z
https://github.com/huggingface/datasets/issues/6184#issuecomment-1697531202
https://api.github.com/repos/huggingface/datasets/issues/6184
IC_kwDODunzps5lLkVC
https://api.github.com/repos/huggingface/datasets/issues/comments/1697531202/reactions
2023-08-29T14:15:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1697531202
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
I understand this may be a limitation of an upstream tool, but for a user for datasets this is very annoying, as when you have dozens of different datasets with different preprocessing functions you can't really move them all into the same file. It may be worth seeing if there is a way to specialize the dependency (eg. subclass it) and enforce behaviors that makes sense for your product.

I was able to work around this for now by setting `__module__ = None`. If such workarounds are required for now it may be better to document it somewhere than a single obscure issue from a long time ago.

As this is a duplicate issue I'm closing it.

I have another issue with the cache https://github.com/huggingface/datasets/issues/6179 can you take a look?
2023-08-29T20:56:49Z
https://github.com/huggingface/datasets/issues/6184#issuecomment-1698129591
https://api.github.com/repos/huggingface/datasets/issues/6184
IC_kwDODunzps5lN2a3
https://api.github.com/repos/huggingface/datasets/issues/comments/1698129591/reactions
2023-08-29T20:57:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1698129591
https://avatars.githubusercontent.com/u/511073?v=4
https://api.github.com/users/jonathanasdf/events{/privacy}
https://api.github.com/users/jonathanasdf/followers
https://api.github.com/users/jonathanasdf/following{/other_user}
https://api.github.com/users/jonathanasdf/gists{/gist_id}
https://github.com/jonathanasdf
jonathanasdf
MDQ6VXNlcjUxMTA3Mw==
https://api.github.com/users/jonathanasdf/orgs
https://api.github.com/users/jonathanasdf/received_events
https://api.github.com/users/jonathanasdf/repos
https://api.github.com/users/jonathanasdf/starred{/owner}{/repo}
https://api.github.com/users/jonathanasdf/subscriptions
User
https://api.github.com/users/jonathanasdf
public
```python
# dataset.py
import os
import datasets

if not os.path.exists('/tmp/test.json'):
  with open('/tmp/test.json', 'w') as file:
    file.write('[{"text": "hello"}]')

def transform(example):
  text = example['text']
  # text += ' world'
  return {'text': text}

data = datasets.load_dataset('json', data_files=['/tmp/test.json'], split='train')
data = data.map(transform)
```

```python
# test.py
import dataset
print(next(iter(dataset.data)))
```

Initialize cache
```
python3 test.py
# {'text': 'hello'}
```

Edit dataset.py and uncomment the commented line, run again
```
python3 test.py
# {'text': 'hello'}
# expected: {'text': 'hello world'}
```

Clear cache and run again
```
rm -rf ~/.cache/huggingface/datasets/*
python3 test.py
# {'text': 'hello world'}
```


If instead the two files are combined, then changes to the function are detected correctly. But it's expected when working on any realistic codebase that things will be modularized into separate files.