Caching doesn't work for map (non-deterministic)
MEMBER
Thanks for reporting !

To store the cache file, we compute a hash of the function given in `.map`, using our own hashing function.
The hash doesn't seem to stay the same over sessions for the tokenizer.
Apparently this is because of the regex at `tokenizer.pat` is not well supported by our hashing function.

I'm working on a fix
2020-08-13T08:44:39Z
https://github.com/huggingface/datasets/issues/501#issuecomment-673348149
https://api.github.com/repos/huggingface/datasets/issues/501
MDEyOklzc3VlQ29tbWVudDY3MzM0ODE0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/673348149/reactions
2020-08-13T08:44:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/673348149
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks everyone. Works great now.
2020-08-24T16:35:00Z
https://github.com/huggingface/datasets/issues/501#issuecomment-679236076
https://api.github.com/repos/huggingface/datasets/issues/501
MDEyOklzc3VlQ29tbWVudDY3OTIzNjA3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/679236076/reactions
2020-08-24T16:35:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/679236076
https://avatars.githubusercontent.com/u/8149933?v=4
https://api.github.com/users/wulu473/events{/privacy}
https://api.github.com/users/wulu473/followers
https://api.github.com/users/wulu473/following{/other_user}
https://api.github.com/users/wulu473/gists{/gist_id}
https://github.com/wulu473
wulu473
MDQ6VXNlcjgxNDk5MzM=
https://api.github.com/users/wulu473/orgs
https://api.github.com/users/wulu473/received_events
https://api.github.com/users/wulu473/repos
https://api.github.com/users/wulu473/starred{/owner}{/repo}
https://api.github.com/users/wulu473/subscriptions
User
https://api.github.com/users/wulu473
public
NONE
Hi. I believe the fix was for the nlp library. Is there a solution to handle compiled regex expressions in .map() with the caching. I want to run a simple regex pattern on a big dataset, but I am running into the issue of compiled expression not being cached. 

Instead of opening a new issue, I thought I would put my query here. Let me know if a new issue would be more suitable. Thanks
2022-08-04T14:37:18Z
https://github.com/huggingface/datasets/issues/501#issuecomment-1205347747
https://api.github.com/repos/huggingface/datasets/issues/501
IC_kwDODunzps5H2CWj
https://api.github.com/repos/huggingface/datasets/issues/comments/1205347747/reactions
2022-08-04T14:37:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1205347747
https://avatars.githubusercontent.com/u/1519077?v=4
https://api.github.com/users/MaveriQ/events{/privacy}
https://api.github.com/users/MaveriQ/followers
https://api.github.com/users/MaveriQ/following{/other_user}
https://api.github.com/users/MaveriQ/gists{/gist_id}
https://github.com/MaveriQ
MaveriQ
MDQ6VXNlcjE1MTkwNzc=
https://api.github.com/users/MaveriQ/orgs
https://api.github.com/users/MaveriQ/received_events
https://api.github.com/users/MaveriQ/repos
https://api.github.com/users/MaveriQ/starred{/owner}{/repo}
https://api.github.com/users/MaveriQ/subscriptions
User
https://api.github.com/users/MaveriQ
public
COLLABORATOR
Hi @MaveriQ! This fix is also included in the `datasets` library. Can you provide a reproducer?
2022-08-08T11:02:23Z
https://github.com/huggingface/datasets/issues/501#issuecomment-1207977348
https://api.github.com/repos/huggingface/datasets/issues/501
IC_kwDODunzps5IAEWE
https://api.github.com/repos/huggingface/datasets/issues/comments/1207977348/reactions
2022-08-08T11:02:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1207977348
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
The caching functionality doesn't work reliably when tokenizing a dataset. Here's a small example to reproduce it. 

```python
import nlp
import transformers

def main():
    ds = nlp.load_dataset("reddit", split="train[:500]")

    tokenizer = transformers.AutoTokenizer.from_pretrained("gpt2")

    def convert_to_features(example_batch):
        input_str = example_batch["body"]
        encodings = tokenizer(input_str, add_special_tokens=True, truncation=True)
        return encodings

    ds = ds.map(convert_to_features, batched=True)

if __name__ == "__main__":
    main()
```

Roughly 3/10 times, this example recomputes the tokenization.

Is this expected behaviour?