Failure to hash function when using .map() 
MEMBER
Hi ! `enc` is not hashable:
```python
import tiktoken
from datasets.fingerprint import Hasher

enc = tiktoken.get_encoding("gpt2")
Hasher.hash(enc)
# raises TypeError: cannot pickle 'builtins.CoreBPE' object
```
It happens because it's not picklable, and because of that it's not possible to cache the result of `map`, hence the warning message.

You can find more details about caching here: https://huggingface.co/docs/datasets/about_cache

You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument.
Or disable caching using
```python
import datasets
datasets.disable_caching()
```
2023-02-16T11:02:56Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1432907917
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5VaHCN
https://api.github.com/repos/huggingface/datasets/issues/comments/1432907917/reactions
2023-02-16T11:02:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1432907917
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq Thank you for the explanation and advice. Will relay all of this to the repo where this (non)issue arose. 

Great job with huggingface! 
2023-02-16T14:56:41Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1433214384
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5VbR2w
https://api.github.com/repos/huggingface/datasets/issues/comments/1433214384/reactions
2023-02-16T14:56:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1433214384
https://avatars.githubusercontent.com/u/6916056?v=4
https://api.github.com/users/venzen/events{/privacy}
https://api.github.com/users/venzen/followers
https://api.github.com/users/venzen/following{/other_user}
https://api.github.com/users/venzen/gists{/gist_id}
https://github.com/venzen
venzen
MDQ6VXNlcjY5MTYwNTY=
https://api.github.com/users/venzen/orgs
https://api.github.com/users/venzen/received_events
https://api.github.com/users/venzen/repos
https://api.github.com/users/venzen/starred{/owner}{/repo}
https://api.github.com/users/venzen/subscriptions
User
https://api.github.com/users/venzen
public
MEMBER
We made tiktoken tokenizers hashable in #5552, which is included in today's release `datasets==2.10.0`
2023-02-22T13:11:14Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1439992846
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5V1IwO
https://api.github.com/repos/huggingface/datasets/issues/comments/1439992846/reactions
2023-02-22T13:11:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1439992846
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Just a heads up that when I'm trying to use TikToken along with the a given Dataset `.map()` method, I am still met with the following error :

```
  File "/opt/conda/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/conda/lib/python3.8/pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle 'builtins.CoreBPE' object
```

My current environment is running datasets v2.10.0.
2023-02-28T00:11:39Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1447319265
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5WRFbh
https://api.github.com/repos/huggingface/datasets/issues/comments/1447319265/reactions
2023-02-28T00:11:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1447319265
https://avatars.githubusercontent.com/u/5572998?v=4
https://api.github.com/users/edhenry/events{/privacy}
https://api.github.com/users/edhenry/followers
https://api.github.com/users/edhenry/following{/other_user}
https://api.github.com/users/edhenry/gists{/gist_id}
https://github.com/edhenry
edhenry
MDQ6VXNlcjU1NzI5OTg=
https://api.github.com/users/edhenry/orgs
https://api.github.com/users/edhenry/received_events
https://api.github.com/users/edhenry/repos
https://api.github.com/users/edhenry/starred{/owner}{/repo}
https://api.github.com/users/edhenry/subscriptions
User
https://api.github.com/users/edhenry
public
MEMBER
cc @mariosasko 
2023-02-28T10:21:28Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1447923846
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5WTZCG
https://api.github.com/repos/huggingface/datasets/issues/comments/1447923846/reactions
2023-02-28T10:21:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1447923846
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq @edhenry I am also seeing this, do you have any suggested solution?
2023-05-18T18:57:42Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1553499107
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5cmIPj
https://api.github.com/repos/huggingface/datasets/issues/comments/1553499107/reactions
2023-05-18T18:57:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1553499107
https://avatars.githubusercontent.com/u/813367?v=4
https://api.github.com/users/moinnadeem/events{/privacy}
https://api.github.com/users/moinnadeem/followers
https://api.github.com/users/moinnadeem/following{/other_user}
https://api.github.com/users/moinnadeem/gists{/gist_id}
https://github.com/moinnadeem
moinnadeem
MDQ6VXNlcjgxMzM2Nw==
https://api.github.com/users/moinnadeem/orgs
https://api.github.com/users/moinnadeem/received_events
https://api.github.com/users/moinnadeem/repos
https://api.github.com/users/moinnadeem/starred{/owner}{/repo}
https://api.github.com/users/moinnadeem/subscriptions
User
https://api.github.com/users/moinnadeem
public
MEMBER
With which `datasets` version ? Can you try to udpate ?
2023-05-19T12:59:43Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1554535561
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5cqFSJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1554535561/reactions
2023-05-19T12:59:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1554535561
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq @edhenry I am on datasets version `'2.12.0'. I see the same `TypeError: cannot pickle 'builtins.CoreBPE' object` that others are seeing.
2023-05-22T20:02:16Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1557884159
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5c22z_
https://api.github.com/repos/huggingface/datasets/issues/comments/1557884159/reactions
2023-05-22T20:02:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1557884159
https://avatars.githubusercontent.com/u/813367?v=4
https://api.github.com/users/moinnadeem/events{/privacy}
https://api.github.com/users/moinnadeem/followers
https://api.github.com/users/moinnadeem/following{/other_user}
https://api.github.com/users/moinnadeem/gists{/gist_id}
https://github.com/moinnadeem
moinnadeem
MDQ6VXNlcjgxMzM2Nw==
https://api.github.com/users/moinnadeem/orgs
https://api.github.com/users/moinnadeem/received_events
https://api.github.com/users/moinnadeem/repos
https://api.github.com/users/moinnadeem/starred{/owner}{/repo}
https://api.github.com/users/moinnadeem/subscriptions
User
https://api.github.com/users/moinnadeem
public
NONE
I am able to reproduce this on datasets 2.14.2. The `datasets.disable_caching()` doesn't work around it.

@lhoestq - you might want to reopen this issue. Because of this issue folks won't be able run Karpathy's NanoGPT :(.
2023-08-03T04:01:46Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1663257492
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5jI0uU
https://api.github.com/repos/huggingface/datasets/issues/comments/1663257492/reactions
2023-08-03T04:13:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1663257492
https://avatars.githubusercontent.com/u/2096835?v=4
https://api.github.com/users/sytelus/events{/privacy}
https://api.github.com/users/sytelus/followers
https://api.github.com/users/sytelus/following{/other_user}
https://api.github.com/users/sytelus/gists{/gist_id}
https://github.com/sytelus
sytelus
MDQ6VXNlcjIwOTY4MzU=
https://api.github.com/users/sytelus/orgs
https://api.github.com/users/sytelus/received_events
https://api.github.com/users/sytelus/repos
https://api.github.com/users/sytelus/starred{/owner}{/repo}
https://api.github.com/users/sytelus/subscriptions
User
https://api.github.com/users/sytelus
public
NONE
update： temporarily solved the problem by setting
```
--preprocess_num_workers  1
```

-------------
I  have met the same problem, here is my env:
```
datasets                      2.14.4
transformers                  4.31.0
tiktoken                      0.4.0
torch                         1.13.1
```
2023-08-10T08:19:12Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1672776314
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5jtIp6
https://api.github.com/repos/huggingface/datasets/issues/comments/1672776314/reactions
2023-08-10T08:37:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1672776314
https://avatars.githubusercontent.com/u/14243491?v=4
https://api.github.com/users/mengban/events{/privacy}
https://api.github.com/users/mengban/followers
https://api.github.com/users/mengban/following{/other_user}
https://api.github.com/users/mengban/gists{/gist_id}
https://github.com/mengban
mengban
MDQ6VXNlcjE0MjQzNDkx
https://api.github.com/users/mengban/orgs
https://api.github.com/users/mengban/received_events
https://api.github.com/users/mengban/repos
https://api.github.com/users/mengban/starred{/owner}{/repo}
https://api.github.com/users/mengban/subscriptions
User
https://api.github.com/users/mengban
public
COLLABORATOR
@mengban I cannot reproduce the issue even with these versions installed. It would help if you could provide info about your system and the `pip list` output.
2023-08-16T16:43:41Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1680940954
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5kMR-a
https://api.github.com/repos/huggingface/datasets/issues/comments/1680940954/reactions
2023-08-16T16:43:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1680940954
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
@mariosasko Please take a look at this
```python
from typing import Any
from datasets import Dataset
import tiktoken

dataset = Dataset.from_list([{"n": str(i)} for i in range(20)])
enc = tiktoken.get_encoding("gpt2")


class A:
    tokenizer = enc #tiktoken.get_encoding("gpt2")

    def __call__(self, example) -> Any:
        ids = self.tokenizer.encode(example["n"])
        example["len"] = len(ids)
        return example

a = A()

def process(example):
    ids = a.tokenizer.encode(example["n"])
    example["len"] = len(ids)
    return example

# success
tokenized = dataset.map(process, desc="tiktoken", num_proc=2)

# raise TypeError: cannot pickle 'builtins.CoreBPE' object
tokenized = dataset.map(a, desc="tiktoken", num_proc=2)
```

pip list
```
datasets                      2.14.4
tiktoken                      0.4.0
```
2023-08-17T08:01:04Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1681820932
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5kPo0E
https://api.github.com/repos/huggingface/datasets/issues/comments/1681820932/reactions
2023-08-17T08:01:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1681820932
https://avatars.githubusercontent.com/u/32546978?v=4
https://api.github.com/users/maxwellzh/events{/privacy}
https://api.github.com/users/maxwellzh/followers
https://api.github.com/users/maxwellzh/following{/other_user}
https://api.github.com/users/maxwellzh/gists{/gist_id}
https://github.com/maxwellzh
maxwellzh
MDQ6VXNlcjMyNTQ2OTc4
https://api.github.com/users/maxwellzh/orgs
https://api.github.com/users/maxwellzh/received_events
https://api.github.com/users/maxwellzh/repos
https://api.github.com/users/maxwellzh/starred{/owner}{/repo}
https://api.github.com/users/maxwellzh/subscriptions
User
https://api.github.com/users/maxwellzh
public
COLLABORATOR
Thanks @maxwellzh! Our `Hasher` works with this snippet, but the problem is running multiprocessing with a non-serializable `tiktoken.Encoding` object.

Inserting the following code before the `map` should fix this:
```python
import copyreg

def pickle_Encoding(enc):
  return (functools.partial(tiktoken.core.Encoding, enc.name, pat_str=enc._pat_str, mergeable_ranks=enc._mergeable_ranks, special_tokens=enc._special_tokens), ())

copyreg.pickle(tiktoken.core.Encoding, pickle_Encoding)
```

But the best fix would be implementing `__reduce__` for `tiktoken.Encoding` or `tiktoken.CoreBPE`. If I find time, I'll try to fix this in the `tiktoken` repo.
2023-08-17T13:41:31Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1682309347
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5kRgDj
https://api.github.com/repos/huggingface/datasets/issues/comments/1682309347/reactions
2023-08-17T13:41:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1682309347
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
I think the right way to fix this would be to have new tokenizer instance for each process. This applies to many other tokenizers that don't support multi-process or have bugs. To do this, first define tokenizer factory class like this:

```
    class TikTokenFactory:
        def __init__(self):
            self._enc = None
            self.eot_token = None

        def encode_ordinary(self, text):
            if self._enc is None:
                self._enc = tiktoken.get_encoding("gpt2")
                self.eot_token = self._enc.eot_token
            return self._enc.encode_ordinary(text)
```

Now use this in `.map()` like this:

```
    # tokenize the dataset
    tokenized = dataset.map(
        partial(process, TikTokenFactory()),
        remove_columns=['text'],
        desc="tokenizing the splits",
        num_proc=max(1, cpu_count()//2),
    )
```

A full working example is here: https://github.com/sytelus/nanoGPT/blob/refactor/nanogpt_common/hf_data_prepare.py
2023-09-08T21:06:00Z
https://github.com/huggingface/datasets/issues/5536#issuecomment-1712221963
https://api.github.com/repos/huggingface/datasets/issues/5536
IC_kwDODunzps5mDm8L
https://api.github.com/repos/huggingface/datasets/issues/comments/1712221963/reactions
2023-09-08T21:06:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1712221963
https://avatars.githubusercontent.com/u/2096835?v=4
https://api.github.com/users/sytelus/events{/privacy}
https://api.github.com/users/sytelus/followers
https://api.github.com/users/sytelus/following{/other_user}
https://api.github.com/users/sytelus/gists{/gist_id}
https://github.com/sytelus
sytelus
MDQ6VXNlcjIwOTY4MzU=
https://api.github.com/users/sytelus/orgs
https://api.github.com/users/sytelus/received_events
https://api.github.com/users/sytelus/repos
https://api.github.com/users/sytelus/starred{/owner}{/repo}
https://api.github.com/users/sytelus/subscriptions
User
https://api.github.com/users/sytelus
public
### Describe the bug

_Parameter 'function'=<function process at 0x7f1ec4388af0> of the transform datasets.arrow_dataset.Dataset.\_map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed._

This issue with `.map()` happens for me consistently, as also described in closed issue #4506

Dataset indices can be individually serialized using dill and pickle without any errors.  I'm using tiktoken to encode in the function passed to map(). Similarly, indices can be individually encoded without error.

### Steps to reproduce the bug

```py
from datasets import load_dataset
import tiktoken

dataset = load_dataset("stas/openwebtext-10k")

enc = tiktoken.get_encoding("gpt2")

tokenized = dataset.map(
    process,
    remove_columns=['text'],
    desc="tokenizing the OWT splits",
)

def process(example):
        ids = enc.encode(example['text'])
        ids.append(enc.eot_token)
        out = {'ids': ids, 'len': len(ids)}
        return out
```

### Expected behavior

Should encode simple text objects.

### Environment info


Python versions tried: both 3.8 and 3.10.10
`PYTHONUTF8=1` as env variable

Datasets tried: 
- stas/openwebtext-10k
- rotten_tomatoes
- local text file

OS: Ubuntu Linux 20.04

Package versions:
- torch 1.13.1
- dill 0.3.4 (if using 0.3.6 - same issue)
- datasets 2.9.0
- tiktoken 0.2.0

