`generate_random_fingerprint()` deterministic with 🤗Transformers' `set_seed()`
NONE
I dug into what I believe is the root of this issue and added a repro in my comment. If this is better addressed as a cross-team issue, let me know and I can open an issue in the Transformers repo
2021-08-10T17:33:38Z
https://github.com/huggingface/datasets/issues/2775#issuecomment-896179404
https://api.github.com/repos/huggingface/datasets/issues/2775
IC_kwDODunzps41apzM
https://api.github.com/repos/huggingface/datasets/issues/comments/896179404/reactions
2021-08-10T17:33:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/896179404
https://avatars.githubusercontent.com/u/1170062?v=4
https://api.github.com/users/mbforbes/events{/privacy}
https://api.github.com/users/mbforbes/followers
https://api.github.com/users/mbforbes/following{/other_user}
https://api.github.com/users/mbforbes/gists{/gist_id}
https://github.com/mbforbes
mbforbes
MDQ6VXNlcjExNzAwNjI=
https://api.github.com/users/mbforbes/orgs
https://api.github.com/users/mbforbes/received_events
https://api.github.com/users/mbforbes/repos
https://api.github.com/users/mbforbes/starred{/owner}{/repo}
https://api.github.com/users/mbforbes/subscriptions
User
https://api.github.com/users/mbforbes
public
MEMBER
Hi !

IMO we shouldn't try to modify `set_seed` from transformers but maybe make `datasets` have its own RNG just to generate random fingerprints.

Any opinion on this @LysandreJik ?
2021-08-23T17:07:54Z
https://github.com/huggingface/datasets/issues/2775#issuecomment-903955529
https://api.github.com/repos/huggingface/datasets/issues/2775
IC_kwDODunzps414URJ
https://api.github.com/repos/huggingface/datasets/issues/comments/903955529/reactions
2021-08-23T17:07:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/903955529
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Yes, this sounds good @lhoestq 
2021-08-26T08:30:54Z
https://github.com/huggingface/datasets/issues/2775#issuecomment-906205333
https://api.github.com/repos/huggingface/datasets/issues/2775
IC_kwDODunzps42A5iV
https://api.github.com/repos/huggingface/datasets/issues/comments/906205333/reactions
2021-08-26T08:30:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/906205333
https://avatars.githubusercontent.com/u/30755778?v=4
https://api.github.com/users/LysandreJik/events{/privacy}
https://api.github.com/users/LysandreJik/followers
https://api.github.com/users/LysandreJik/following{/other_user}
https://api.github.com/users/LysandreJik/gists{/gist_id}
https://github.com/LysandreJik
LysandreJik
MDQ6VXNlcjMwNzU1Nzc4
https://api.github.com/users/LysandreJik/orgs
https://api.github.com/users/LysandreJik/received_events
https://api.github.com/users/LysandreJik/repos
https://api.github.com/users/LysandreJik/starred{/owner}{/repo}
https://api.github.com/users/LysandreJik/subscriptions
User
https://api.github.com/users/LysandreJik
public
## Describe the bug

**Update:** I dug into this to try to reproduce the underlying issue, and I believe it's that `set_seed()` from the `transformers` library makes the "random" fingerprint identical each time. I believe this is still a bug, because `datasets` is used exactly this way in `transformers` after `set_seed()` has been called, and I think that using `set_seed()` is a standard procedure to aid reproducibility. I've added more details to reproduce this below.

Hi there! I'm using my own local dataset and custom preprocessing function. My preprocessing function seems to be unpickle-able, perhaps because it is from a closure (will debug this separately). I get this warning, which is expected:

https://github.com/huggingface/datasets/blob/450b9174765374111e5c6daab0ed294bc3d9b639/src/datasets/fingerprint.py#L260-L265

However, what's not expected is that the `datasets` actually _does_ seem to cache and reuse this dataset between runs! After that line, the next thing that's logged looks like:

```text
 Loading cached processed dataset at /home/xxx/.cache/huggingface/datasets/csv/default-xxx/0.0.0/xxx/cache-xxx.arrow
```

The path is exactly the same each run (e.g., last 26 runs).

This becomes a problem because I'll pass in the `--max_eval_samples` flag to the HuggingFace example script I'm running off of ([run_swag.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/multiple-choice/run_swag.py)).  The fact that the cached dataset is reused means this flag gets ignored. I'll try to load 100 examples, and it will load the full cached 1,000,000.

I think that

https://github.com/huggingface/datasets/blob/450b9174765374111e5c6daab0ed294bc3d9b639/src/datasets/fingerprint.py#L248

... is actually consistent because randomness is being controlled in HuggingFace/Transformers for reproducibility. I've added a demo of this below.

## Steps to reproduce the bug

```python
# Contents of print_fingerprint.py
from transformers import set_seed
from datasets.fingerprint import generate_random_fingerprint
set_seed(42)
print(generate_random_fingerprint())
```

```bash
for i in {0..10}; do
    python print_fingerprint.py
done

1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
1c80317fa3b1799d
```

## Expected results
After the "random hash" warning is emitted, a random hash is generated, and no outdated cached datasets are reused.

## Actual results
After the "random hash" warning is emitted, an identical hash is generated each time, and an outdated cached dataset is reused each run.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.0
- Platform: Linux-5.8.0-1038-gcp-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1