Make save function use deterministic global vars order
NONE
Sorry, asking for help here, but the dill thread stop around 2013. Is it possible to use dill deterministically? I tried to monkeypatch the solution presented here into dill, but I suppose it requires forking their project.
2021-11-26T19:07:58Z
https://github.com/huggingface/datasets/pull/819#issuecomment-980314068
https://api.github.com/repos/huggingface/datasets/issues/819
IC_kwDODunzps46bmfU
https://api.github.com/repos/huggingface/datasets/issues/comments/980314068/reactions
2021-11-26T19:07:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/980314068
https://avatars.githubusercontent.com/u/3620506?v=4
https://api.github.com/users/davips/events{/privacy}
https://api.github.com/users/davips/followers
https://api.github.com/users/davips/following{/other_user}
https://api.github.com/users/davips/gists{/gist_id}
https://github.com/davips
davips
MDQ6VXNlcjM2MjA1MDY=
https://api.github.com/users/davips/orgs
https://api.github.com/users/davips/received_events
https://api.github.com/users/davips/repos
https://api.github.com/users/davips/starred{/owner}{/repo}
https://api.github.com/users/davips/subscriptions
User
https://api.github.com/users/davips
public
MEMBER
Hi ! What we did was to subclass `dill`'s Pickler to fix the non-deterministic behaviors, and it's been working fine. A fork should also do the job
2021-11-30T13:34:09Z
https://github.com/huggingface/datasets/pull/819#issuecomment-982641730
https://api.github.com/repos/huggingface/datasets/issues/819
IC_kwDODunzps46kexC
https://api.github.com/repos/huggingface/datasets/issues/comments/982641730/reactions
2021-11-30T13:34:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/982641730
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
The `dumps` function need to be deterministic for the caching mechanism.
However in #816 I noticed that one of dill's method to recursively check the globals of a function may return the globals in different orders each time it's used. To fix that I sort the globals by key in the `globs` dictionary.
I had to add a rectified `save_function` to the saving functions registry of the Pickler to make it work.

This should fix #816 