Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0
MEMBER
Hi ! We can bump `dill` to 0.3.9 if we make sure it's deterministic and doesn't break the caching mechanism in `datasets`.

Would you be interested in opening a PR ? Then we can run the CI to see if it works
2025-04-15T12:43:33Z
https://github.com/huggingface/datasets/issues/7510#issuecomment-2804928953
https://api.github.com/repos/huggingface/datasets/issues/7510
IC_kwDODunzps6nL9G5
https://api.github.com/repos/huggingface/datasets/issues/comments/2804928953/reactions
2025-04-15T12:43:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2804928953
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi!. Yeah I can do it. Should I make any changes besides dill versions?
2025-04-16T06:05:00Z
https://github.com/huggingface/datasets/issues/7510#issuecomment-2808470854
https://api.github.com/repos/huggingface/datasets/issues/7510
IC_kwDODunzps6nZd1G
https://api.github.com/repos/huggingface/datasets/issues/comments/2808470854/reactions
2025-04-16T06:05:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2808470854
https://avatars.githubusercontent.com/u/98061329?v=4
https://api.github.com/users/JGrel/events{/privacy}
https://api.github.com/users/JGrel/followers
https://api.github.com/users/JGrel/following{/other_user}
https://api.github.com/users/JGrel/gists{/gist_id}
https://github.com/JGrel
JGrel
U_kgDOBdhMEQ
https://api.github.com/users/JGrel/orgs
https://api.github.com/users/JGrel/received_events
https://api.github.com/users/JGrel/repos
https://api.github.com/users/JGrel/starred{/owner}{/repo}
https://api.github.com/users/JGrel/subscriptions
User
https://api.github.com/users/JGrel
public
MEMBER
There are probably some usage of internal functions from `dill` that we'll need to update in `datasets`

If you run `pytest tests/test_fingerprint.py` you should already have a good idea of what works and what doesn't.
But feel free to open a PR anyway, this way we can run the full CI and see the results

2025-04-16T09:30:16Z
https://github.com/huggingface/datasets/issues/7510#issuecomment-2808970142
https://api.github.com/repos/huggingface/datasets/issues/7510
IC_kwDODunzps6nbXue
https://api.github.com/repos/huggingface/datasets/issues/comments/2808970142/reactions
2025-04-16T09:30:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2808970142
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, sorry for no response from my side. I will try to do it today.
2025-04-23T07:04:31Z
https://github.com/huggingface/datasets/issues/7510#issuecomment-2823275462
https://api.github.com/repos/huggingface/datasets/issues/7510
IC_kwDODunzps6oR8PG
https://api.github.com/repos/huggingface/datasets/issues/comments/2823275462/reactions
2025-04-23T07:04:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2823275462
https://avatars.githubusercontent.com/u/98061329?v=4
https://api.github.com/users/JGrel/events{/privacy}
https://api.github.com/users/JGrel/followers
https://api.github.com/users/JGrel/following{/other_user}
https://api.github.com/users/JGrel/gists{/gist_id}
https://github.com/JGrel
JGrel
U_kgDOBdhMEQ
https://api.github.com/users/JGrel/orgs
https://api.github.com/users/JGrel/received_events
https://api.github.com/users/JGrel/repos
https://api.github.com/users/JGrel/starred{/owner}{/repo}
https://api.github.com/users/JGrel/subscriptions
User
https://api.github.com/users/JGrel
public
NONE
Created pull request: [LINK](https://github.com/huggingface/datasets/pull/7535)
Tried to run tests by using command you have send and got few errors:

![Image](https://github.com/user-attachments/assets/acbf1feb-4dd1-416e-a118-c91abe0d188b)
2025-04-24T19:49:29Z
https://github.com/huggingface/datasets/issues/7510#issuecomment-2828697827
https://api.github.com/repos/huggingface/datasets/issues/7510
IC_kwDODunzps6omoDj
https://api.github.com/repos/huggingface/datasets/issues/comments/2828697827/reactions
2025-04-24T19:49:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2828697827
https://avatars.githubusercontent.com/u/98061329?v=4
https://api.github.com/users/JGrel/events{/privacy}
https://api.github.com/users/JGrel/followers
https://api.github.com/users/JGrel/following{/other_user}
https://api.github.com/users/JGrel/gists{/gist_id}
https://github.com/JGrel
JGrel
U_kgDOBdhMEQ
https://api.github.com/users/JGrel/orgs
https://api.github.com/users/JGrel/received_events
https://api.github.com/users/JGrel/repos
https://api.github.com/users/JGrel/starred{/owner}{/repo}
https://api.github.com/users/JGrel/subscriptions
User
https://api.github.com/users/JGrel
public
MEMBER
Thanks for running the test ! So it appears we have two issues to fix:
1. 'log' is not defined: it seems an internal `dill` function has disappeared, so we should adapt the `datasets` code that was using it
2. there are some hashes mismatches, which means `dill` doesn't seem to output the same dump when passed the same ipython function twice, or the same function but located at a different line in a python file
2025-05-19T14:54:03Z
https://github.com/huggingface/datasets/issues/7510#issuecomment-2891336903
https://api.github.com/repos/huggingface/datasets/issues/7510
IC_kwDODunzps6sVkzH
https://api.github.com/repos/huggingface/datasets/issues/comments/2891336903/reactions
2025-05-19T14:54:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2891336903
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Describe the bug

Datasets 2.18.0 - 3.5.0 has a dependency on dill < 0.3.9. This causes errors with dill >= 0.3.9.

Could you please take a look into it and make it compatible?

### Steps to reproduce the bug

1. Install setuptools >= 2.18.0
2. Install dill >=0.3.9
3. Run pip check
4. Output:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires dill<0.3.9,>=0.3.0, but you have dill 0.3.9 which is incompatible.

### Expected behavior

Pip install both libraries without any errors

### Environment info

Datasets version: 2.18 - 3.5
Python: 3.11