Consistent caching between python and jupyter
MEMBER
Hi ! Maybe it's possible to have a consistent hash for a function defined in `__main__` and a function define in a notebook.

However for functions imported from another location, pickle uses the location to identify the code, so in that case we can't do much I believe.

Would it be ok for you if we only try to do this for functions in `__main__` / jupyter ?

If you'd like to contribute, you can read this part of the code and let me know if you have questions:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/utils/py_utils.py#L617-L643

I think the key here would be to also ignore the "co_filename" of functions defined in `__main__`
2022-10-25T09:47:29Z
https://github.com/huggingface/datasets/issues/5157#issuecomment-1290281696
https://api.github.com/repos/huggingface/datasets/issues/5157
IC_kwDODunzps5M6CLg
https://api.github.com/repos/huggingface/datasets/issues/comments/1290281696/reactions
2022-10-25T09:47:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1290281696
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Seems like a good solution, I will start a PR and see if I understood the changes needed. Thanks!
2022-10-27T02:14:09Z
https://github.com/huggingface/datasets/issues/5157#issuecomment-1292872558
https://api.github.com/repos/huggingface/datasets/issues/5157
IC_kwDODunzps5ND6tu
https://api.github.com/repos/huggingface/datasets/issues/comments/1292872558/reactions
2022-10-27T02:14:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1292872558
https://avatars.githubusercontent.com/u/32967787?v=4
https://api.github.com/users/gpucce/events{/privacy}
https://api.github.com/users/gpucce/followers
https://api.github.com/users/gpucce/following{/other_user}
https://api.github.com/users/gpucce/gists{/gist_id}
https://github.com/gpucce
gpucce
MDQ6VXNlcjMyOTY3Nzg3
https://api.github.com/users/gpucce/orgs
https://api.github.com/users/gpucce/received_events
https://api.github.com/users/gpucce/repos
https://api.github.com/users/gpucce/starred{/owner}{/repo}
https://api.github.com/users/gpucce/subscriptions
User
https://api.github.com/users/gpucce
public
### Feature request

I hope this is not my mistake, currently if I use `load_dataset` from a python session on a custom dataset to do the preprocessing, it will be saved in the cache and in other python sessions it will be loaded from the cache, however calling the same from a jupyter notebook does not work, meaning the preprocessing starts from scratch.

If adjusting the hashes is impossible, is there a way to manually set dataset fingerprint to "force" this behaviour?

### Motivation

If this is not already the case and I am doing something wrong, it would be useful to have the two fingerprints consistent so one can create the dataset once and then try small things on jupyter without preprocessing everything again.


### Your contribution

I am happy to try a PR if you give me some pointers where the changes should happen