Cached dataset not loaded
CONTRIBUTOR
It actually seems to happen all the time in above configuration:
* the function `filter_by_duration` correctly loads cached processed dataset
* the function `prepare_dataset` is always reexecuted

I end up solving the issue by saving to disk my dataset at the end but I'm still wondering if it's a bug or limitation here.
2021-05-28T13:55:09Z
https://github.com/huggingface/datasets/issues/2415#issuecomment-850436771
https://api.github.com/repos/huggingface/datasets/issues/2415
MDEyOklzc3VlQ29tbWVudDg1MDQzNjc3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/850436771/reactions
2021-05-28T13:55:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/850436771
https://avatars.githubusercontent.com/u/715491?v=4
https://api.github.com/users/borisdayma/events{/privacy}
https://api.github.com/users/borisdayma/followers
https://api.github.com/users/borisdayma/following{/other_user}
https://api.github.com/users/borisdayma/gists{/gist_id}
https://github.com/borisdayma
borisdayma
MDQ6VXNlcjcxNTQ5MQ==
https://api.github.com/users/borisdayma/orgs
https://api.github.com/users/borisdayma/received_events
https://api.github.com/users/borisdayma/repos
https://api.github.com/users/borisdayma/starred{/owner}{/repo}
https://api.github.com/users/borisdayma/subscriptions
User
https://api.github.com/users/borisdayma
public
MEMBER
Hi ! The hash used for caching `map` results is the fingerprint of the resulting dataset. It is computed using three things:
- the old fingerprint of the dataset
- the hash of the function
- the hash of the other parameters passed to `map`

You can compute the hash of your function (or any python object) with
```python
from datasets.fingerprint import Hasher

my_func = lambda x: x + 1
print(Hasher.hash(my_func))
```

If `prepare_dataset` is always executed, maybe this is because your `processor` has a different hash each time you want to execute it.
2021-06-01T08:46:55Z
https://github.com/huggingface/datasets/issues/2415#issuecomment-851948859
https://api.github.com/repos/huggingface/datasets/issues/2415
MDEyOklzc3VlQ29tbWVudDg1MTk0ODg1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/851948859/reactions
2021-06-01T08:47:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/851948859
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> If `prepare_dataset` is always executed, maybe this is because your `processor` has a different hash each time you want to execute it.

Yes I think that was the issue.

For the hash of the function:
* does it consider just the name or the actual code of the function
* does it consider variables that are not passed explicitly as parameters to the functions (such as the processor here)
2021-06-02T03:27:53Z
https://github.com/huggingface/datasets/issues/2415#issuecomment-852687348
https://api.github.com/repos/huggingface/datasets/issues/2415
MDEyOklzc3VlQ29tbWVudDg1MjY4NzM0OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/852687348/reactions
2021-06-02T03:27:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/852687348
https://avatars.githubusercontent.com/u/715491?v=4
https://api.github.com/users/borisdayma/events{/privacy}
https://api.github.com/users/borisdayma/followers
https://api.github.com/users/borisdayma/following{/other_user}
https://api.github.com/users/borisdayma/gists{/gist_id}
https://github.com/borisdayma
borisdayma
MDQ6VXNlcjcxNTQ5MQ==
https://api.github.com/users/borisdayma/orgs
https://api.github.com/users/borisdayma/received_events
https://api.github.com/users/borisdayma/repos
https://api.github.com/users/borisdayma/starred{/owner}{/repo}
https://api.github.com/users/borisdayma/subscriptions
User
https://api.github.com/users/borisdayma
public
MEMBER
> does it consider just the name or the actual code of the function

It looks at the name and the actual code and all variables such as recursively. It uses `dill` to do so, which is based on `pickle`.
Basically the hash is computed using the pickle bytes of your function (computed using `dill` to support most python objects).

> does it consider variables that are not passed explicitly as parameters to the functions (such as the processor here)

Yes it does thanks to recursive pickling.
2021-06-02T10:07:36Z
https://github.com/huggingface/datasets/issues/2415#issuecomment-852898228
https://api.github.com/repos/huggingface/datasets/issues/2415
MDEyOklzc3VlQ29tbWVudDg1Mjg5ODIyOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/852898228/reactions
2021-06-02T10:07:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/852898228
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks for these explanations. I'm closing the issue.
2021-06-02T13:15:47Z
https://github.com/huggingface/datasets/issues/2415#issuecomment-853018368
https://api.github.com/repos/huggingface/datasets/issues/2415
MDEyOklzc3VlQ29tbWVudDg1MzAxODM2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/853018368/reactions
2021-06-02T13:15:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/853018368
https://avatars.githubusercontent.com/u/715491?v=4
https://api.github.com/users/borisdayma/events{/privacy}
https://api.github.com/users/borisdayma/followers
https://api.github.com/users/borisdayma/following{/other_user}
https://api.github.com/users/borisdayma/gists{/gist_id}
https://github.com/borisdayma
borisdayma
MDQ6VXNlcjcxNTQ5MQ==
https://api.github.com/users/borisdayma/orgs
https://api.github.com/users/borisdayma/received_events
https://api.github.com/users/borisdayma/repos
https://api.github.com/users/borisdayma/starred{/owner}{/repo}
https://api.github.com/users/borisdayma/subscriptions
User
https://api.github.com/users/borisdayma
public
## Describe the bug
I have a large dataset (common_voice, english) where I use several map and filter functions.
Sometimes my cached datasets after specific functions are not loaded.
I always use the same arguments, same functions, no seed…

## Steps to reproduce the bug
```python
def filter_by_duration(batch):
    return (
        batch["duration"] <= 10
        and batch["duration"] >= 1
        and len(batch["target_text"]) > 5
    )

def prepare_dataset(batch):
    batch["input_values"] = processor(
        batch["speech"], sampling_rate=batch["sampling_rate"][0]
    ).input_values
    with processor.as_target_processor():
        batch["labels"] = processor(batch["target_text"]).input_ids
    return batch

train_dataset = train_dataset.filter(
    filter_by_duration,
    remove_columns=["duration"],
    num_proc=data_args.preprocessing_num_workers,
)

# PROBLEM HERE -> below function is reexecuted and cache is not loaded
train_dataset = train_dataset.map(
    prepare_dataset,
    remove_columns=train_dataset.column_names,
    batch_size=training_args.per_device_train_batch_size,
    batched=True,
    num_proc=data_args.preprocessing_num_workers,
)

# Later in script
set_caching_enabled(False)
# apply map on trained model to eval/test sets

```

## Expected results
The cached dataset should always be reloaded.

## Actual results
The function is reexecuted.

I have access to cached files `cache-xxxxx.arrow`.
Is there a way I can somehow load manually 2 versions and see how the hash was created for debug purposes (to know if it's an issue with dataset or function)?

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.2
- Platform: Linux-5.8.0-45-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No