AutoTokenizer hash value got change after datasets.map
NONE
This issue was original reported at https://github.com/huggingface/transformers/issues/14931 and It seems like this issue also occur with other AutoClass like AutoFeatureExtractor.
2022-01-27T03:19:48Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1022809627
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps489tYb
https://api.github.com/repos/huggingface/datasets/issues/comments/1022809627/reactions
2022-01-27T03:19:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1022809627
https://avatars.githubusercontent.com/u/13161779?v=4
https://api.github.com/users/tshu-w/events{/privacy}
https://api.github.com/users/tshu-w/followers
https://api.github.com/users/tshu-w/following{/other_user}
https://api.github.com/users/tshu-w/gists{/gist_id}
https://github.com/tshu-w
tshu-w
MDQ6VXNlcjEzMTYxNzc5
https://api.github.com/users/tshu-w/orgs
https://api.github.com/users/tshu-w/received_events
https://api.github.com/users/tshu-w/repos
https://api.github.com/users/tshu-w/starred{/owner}{/repo}
https://api.github.com/users/tshu-w/subscriptions
User
https://api.github.com/users/tshu-w
public
MEMBER
Thanks for moving the issue here !

I wasn't able to reproduce the issue on my env (the hashes stay the same):
```
- `transformers` version: 1.15.0
- `tokenizers` version: 0.10.3
- `datasets` version: 1.18.1
- `dill` version: 0.3.4
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.10
- PyArrow version: 6.0.1
```
However I was able to reproduce it on Google Colab (the hashes end up different):
```
- `transformers` version: 1.15.0
- `tokenizers` version: 0.10.3
- `datasets` version: 1.18.1
- `dill` version: 0.3.4
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0
```
I'll investigate why it doesn't work properly on Google Colab :)
2022-01-27T13:35:21Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1023217215
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps48_Q4_
https://api.github.com/repos/huggingface/datasets/issues/comments/1023217215/reactions
2022-01-27T13:35:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1023217215
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I found the issue: the tokenizer has something inside it that changes.

Before the call, `tokenizer._tokenizer.truncation` is None, and after the call it changes to this for some reason:
```
{'max_length': 512, 'strategy': 'longest_first', 'stride': 0}
```

Does anybody know why calling the tokenizer would change its state this way ? cc @Narsil @SaulLu maybe ?
2022-01-27T14:23:09Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1023262014
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps48_b0-
https://api.github.com/repos/huggingface/datasets/issues/comments/1023262014/reactions
2022-01-27T14:23:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1023262014
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
`tokenizer.encode(..)` does not accept argument like max_length, strategy or stride.

In `tokenizers` you have to modify the tokenizer state by setting various `TruncationParams` (and/or `PaddingParams`).
However, since this is modifying the state, you need to mutably borrow the tokenizer (a rust concept). The key principle is that there can ever be only 1 mutable borrow at a time during the span of the tokenizer lifecycle.

Because of this, if `transformers` blindly set  `TruncationParams` and `PaddingParams` on every call, it would cause the tokenizer to crash (or make the various threads accessing it hang, which is not necessarily better).

In order to avoid that, we decided to handle it this way : https://github.com/huggingface/transformers/pull/12550 . 

Which should explain the state of the tokenizer being modified (hence its hash).

Now for a temporary solution, simply encoding once with the tokenizer should give it it's proper hash (since by default the tokenizer doesn't have this state, looks at the first encoding call, and creates it).

We could try and set these 2 dicts at initialization time, but it wouldn't work if a user modified the tokenizer state later
```python
tokenizer = AutoTokenizer.from_pretrained(..)
tokenizer.truncation_side = "left"
# Now we have a difference between `tokenizer._tokenizer.truncation` and `tokenizer.truncation_side`
```
If we wanted to fix it correctly it would mean mapping every assignation to it's proper location on `tokenizer.{padding/truncation}`

I think it's important to note that we cannot guarantee a tokenizer' hash remains the same if *any* of those parameters are modified through the `.map` function.

Edit: Another option would be to override the default __hash__ function, but I don't know if there's  a sound implementation that could fit.
2022-01-27T14:41:28Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1023280361
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps48_gTp
https://api.github.com/repos/huggingface/datasets/issues/comments/1023280361/reactions
2022-01-27T14:47:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1023280361
https://avatars.githubusercontent.com/u/204321?v=4
https://api.github.com/users/Narsil/events{/privacy}
https://api.github.com/users/Narsil/followers
https://api.github.com/users/Narsil/following{/other_user}
https://api.github.com/users/Narsil/gists{/gist_id}
https://github.com/Narsil
Narsil
MDQ6VXNlcjIwNDMyMQ==
https://api.github.com/users/Narsil/orgs
https://api.github.com/users/Narsil/received_events
https://api.github.com/users/Narsil/repos
https://api.github.com/users/Narsil/starred{/owner}{/repo}
https://api.github.com/users/Narsil/subscriptions
User
https://api.github.com/users/Narsil
public
MEMBER
Thanks a lot for the explanation !
I think if we set these 2 dicts at initialization time it would be amazing already

Shall we open an issue in `transformers` to ask for these dictionaries to be set when the tokenizer is instantiated ?

> Edit: Another option would be to override the default hash function, but I don't know if there's a sound implementation that could fit.

In `datasets` we can easily have custom hashing for objects of the other HF libraries if we want. For example we ignore the cache some tokenizers have. However in this specific case it touches parameters that may change the behavior of the tokenizer itself. I'm not sure the logic that determines how a tokenizer behaves should be in `datasets`
2022-01-27T14:58:30Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1023298151
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps48_kpn
https://api.github.com/repos/huggingface/datasets/issues/comments/1023298151/reactions
2022-01-27T14:58:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1023298151
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
A hack we could have in the `datasets` lib would be to call the tokenizer before hashing it in order to set all its parameters correctly - but it sounds a lot like a hack and I'm not sure this can work in the long run
2022-01-27T14:59:36Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1023299311
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps48_k7v
https://api.github.com/repos/huggingface/datasets/issues/comments/1023299311/reactions
2022-01-27T14:59:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1023299311
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Fully agree with everything you said. 

I think the best course of action is creating an issue in `transformers`. I can start the work on this.
I think the code changes are fairly simple. Making a sound test + not breaking other stuff might be different :D
2022-01-27T15:07:02Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1023308368
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps48_nJQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1023308368/reactions
2022-01-27T15:07:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1023308368
https://avatars.githubusercontent.com/u/204321?v=4
https://api.github.com/users/Narsil/events{/privacy}
https://api.github.com/users/Narsil/followers
https://api.github.com/users/Narsil/following{/other_user}
https://api.github.com/users/Narsil/gists{/gist_id}
https://github.com/Narsil
Narsil
MDQ6VXNlcjIwNDMyMQ==
https://api.github.com/users/Narsil/orgs
https://api.github.com/users/Narsil/received_events
https://api.github.com/users/Narsil/repos
https://api.github.com/users/Narsil/starred{/owner}{/repo}
https://api.github.com/users/Narsil/subscriptions
User
https://api.github.com/users/Narsil
public
NONE
It should be noted that this problem also occurs in other AutoClasses, such as AutoFeatureExtractor, so I don't think handling it in Datasets is a long-term practice either.
2022-01-27T15:07:18Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1023308742
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps48_nPG
https://api.github.com/repos/huggingface/datasets/issues/comments/1023308742/reactions
2022-01-27T15:07:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1023308742
https://avatars.githubusercontent.com/u/13161779?v=4
https://api.github.com/users/tshu-w/events{/privacy}
https://api.github.com/users/tshu-w/followers
https://api.github.com/users/tshu-w/following{/other_user}
https://api.github.com/users/tshu-w/gists{/gist_id}
https://github.com/tshu-w
tshu-w
MDQ6VXNlcjEzMTYxNzc5
https://api.github.com/users/tshu-w/orgs
https://api.github.com/users/tshu-w/received_events
https://api.github.com/users/tshu-w/repos
https://api.github.com/users/tshu-w/starred{/owner}{/repo}
https://api.github.com/users/tshu-w/subscriptions
User
https://api.github.com/users/tshu-w
public
NONE
> I think the best course of action is creating an issue in `transformers`. I can start the work on this.

@Narsil Hi, I reopen this issue in `transformers` https://github.com/huggingface/transformers/issues/14931
2022-01-28T03:20:38Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1023849383
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps49BrOn
https://api.github.com/repos/huggingface/datasets/issues/comments/1023849383/reactions
2022-01-28T03:20:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1023849383
https://avatars.githubusercontent.com/u/13161779?v=4
https://api.github.com/users/tshu-w/events{/privacy}
https://api.github.com/users/tshu-w/followers
https://api.github.com/users/tshu-w/following{/other_user}
https://api.github.com/users/tshu-w/gists{/gist_id}
https://github.com/tshu-w
tshu-w
MDQ6VXNlcjEzMTYxNzc5
https://api.github.com/users/tshu-w/orgs
https://api.github.com/users/tshu-w/received_events
https://api.github.com/users/tshu-w/repos
https://api.github.com/users/tshu-w/starred{/owner}{/repo}
https://api.github.com/users/tshu-w/subscriptions
User
https://api.github.com/users/tshu-w
public
NONE
Here is @Narsil comment from https://github.com/huggingface/transformers/issues/14931#issuecomment-1074981569
> # TL;DR
> Call the function once on a dummy example beforehand will fix it.
> 
> ```python
> tokenizer("Some", "test", truncation=True)
> ```
> 
> # Long answer
> If I remember the last status, it's hard doing anything, since the call itself
> 
> ```python
> tokenizer(example["sentence1"], example["sentence2"], truncation=True)
> ```
> 
> will modify the tokenizer. It's the `truncation=True` that modifies the tokenizer to put it into truncation mode if you will. Calling the tokenizer once with that argument would fix the cache.
> 
> Finding a fix that :
> 
> * Doesn't imply a huge chunk of work on `tokenizers` (with potential loss of performance, and breaking backward compatibility)
> * Doesn't imply `datasets` running a first pass of the loop
> * Doesn't imply `datasets` looking at the map function itself
> * Uses a sound `hash` for this object in `datasets`.
> 
> is IIRC impossible for this use case.
> 
> I can explain a bit more why the first option is not desirable.
> 
> In order to "fix" this for tokenizers, we would need to make `tokenizer(..)` purely without side effects. This means that the "options" of tokenization (like `truncation` and `padding` at least) would have

2022-03-23T08:36:40Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1076087239
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps5AI8nH
https://api.github.com/repos/huggingface/datasets/issues/comments/1076087239/reactions
2022-03-23T08:36:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1076087239
https://avatars.githubusercontent.com/u/13161779?v=4
https://api.github.com/users/tshu-w/events{/privacy}
https://api.github.com/users/tshu-w/followers
https://api.github.com/users/tshu-w/following{/other_user}
https://api.github.com/users/tshu-w/gists{/gist_id}
https://github.com/tshu-w
tshu-w
MDQ6VXNlcjEzMTYxNzc5
https://api.github.com/users/tshu-w/orgs
https://api.github.com/users/tshu-w/received_events
https://api.github.com/users/tshu-w/repos
https://api.github.com/users/tshu-w/starred{/owner}{/repo}
https://api.github.com/users/tshu-w/subscriptions
User
https://api.github.com/users/tshu-w
public
NONE
For me this workaround only works if I don't pass the `num_proc=X` argument to `datasets.map`
2022-08-26T07:47:56Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1228169129
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps5JNF-p
https://api.github.com/repos/huggingface/datasets/issues/comments/1228169129/reactions
2022-08-26T07:47:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1228169129
https://avatars.githubusercontent.com/u/5028974?v=4
https://api.github.com/users/david-waterworth/events{/privacy}
https://api.github.com/users/david-waterworth/followers
https://api.github.com/users/david-waterworth/following{/other_user}
https://api.github.com/users/david-waterworth/gists{/gist_id}
https://github.com/david-waterworth
david-waterworth
MDQ6VXNlcjUwMjg5NzQ=
https://api.github.com/users/david-waterworth/orgs
https://api.github.com/users/david-waterworth/received_events
https://api.github.com/users/david-waterworth/repos
https://api.github.com/users/david-waterworth/starred{/owner}{/repo}
https://api.github.com/users/david-waterworth/subscriptions
User
https://api.github.com/users/david-waterworth
public
NONE
Is there an easy solution for setting both num_proc and padding/truncation for fast tokenizer or caching just not a thing in this case? 
2024-03-11T13:56:14Z
https://github.com/huggingface/datasets/issues/3638#issuecomment-1988504961
https://api.github.com/repos/huggingface/datasets/issues/3638
IC_kwDODunzps52hi2B
https://api.github.com/repos/huggingface/datasets/issues/comments/1988504961/reactions
2024-03-11T13:56:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1988504961
https://avatars.githubusercontent.com/u/26613311?v=4
https://api.github.com/users/DeepCowProductions/events{/privacy}
https://api.github.com/users/DeepCowProductions/followers
https://api.github.com/users/DeepCowProductions/following{/other_user}
https://api.github.com/users/DeepCowProductions/gists{/gist_id}
https://github.com/DeepCowProductions
DeepCowProductions
MDQ6VXNlcjI2NjEzMzEx
https://api.github.com/users/DeepCowProductions/orgs
https://api.github.com/users/DeepCowProductions/received_events
https://api.github.com/users/DeepCowProductions/repos
https://api.github.com/users/DeepCowProductions/starred{/owner}{/repo}
https://api.github.com/users/DeepCowProductions/subscriptions
User
https://api.github.com/users/DeepCowProductions
public
## Describe the bug
AutoTokenizer hash value got change after datasets.map

## Steps to reproduce the bug
1. trash huggingface datasets cache
2. run the following code:
```python
from transformers import AutoTokenizer, BertTokenizer
from datasets import load_dataset
from datasets.fingerprint import Hasher
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

raw_datasets = load_dataset("glue", "mrpc")

print(Hasher.hash(tokenize_function))
print(Hasher.hash(tokenizer))

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

print(Hasher.hash(tokenize_function))
print(Hasher.hash(tokenizer))
```
got
```
Reusing dataset glue (/home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1112.35it/s]
f4976bb4694ebc51
3fca35a1fd4a1251
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  6.96ba/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 15.25ba/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  5.81ba/s]
d32837619b7d7d01
5fd925c82edd62b6
```
3. run raw_datasets.map(tokenize_function, batched=True) again and see some dataset are not using cache.

## Expected results
`AutoTokenizer` work like specific Tokenizer (The hash value don't change after map):
```python
from transformers import AutoTokenizer, BertTokenizer
from datasets import load_dataset
from datasets.fingerprint import Hasher
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

raw_datasets = load_dataset("glue", "mrpc")

print(Hasher.hash(tokenize_function))
print(Hasher.hash(tokenizer))

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

print(Hasher.hash(tokenize_function))
print(Hasher.hash(tokenizer))
```

```
Reusing dataset glue (/home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1091.22it/s]
46d4b31f54153fc7
5b8771afd8d43888
Loading cached processed dataset at /home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad/cache-6b07ff82ae9d5c51.arrow
Loading cached processed dataset at /home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad/cache-af738a6d84f3864b.arrow
Loading cached processed dataset at /home1/wts/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad/cache-531d2a603ba713c1.arrow
46d4b31f54153fc7
5b8771afd8d43888
```


## Environment info
- `datasets` version: 1.18.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 6.0.1
