The current version of the package on github has an error when loading dataset
MEMBER
Thanks for reporting !
Which version of transformers are you using ?
It looks like it doesn't have the PreTrainedTokenizerBase class
2020-09-09T22:31:53Z
https://github.com/huggingface/datasets/issues/598#issuecomment-689857017
https://api.github.com/repos/huggingface/datasets/issues/598
MDEyOklzc3VlQ29tbWVudDY4OTg1NzAxNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/689857017/reactions
2020-09-09T22:31:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/689857017
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I was using transformer 2.9. And I switch to the latest transformer package. Everything works just fine!!

Thanks for helping! I should look more carefully next time. Didn't realize loading the data part requires using tokenizer.

2020-09-09T22:47:47Z
https://github.com/huggingface/datasets/issues/598#issuecomment-689862833
https://api.github.com/repos/huggingface/datasets/issues/598
MDEyOklzc3VlQ29tbWVudDY4OTg2MjgzMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/689862833/reactions
2020-09-09T22:48:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/689862833
https://avatars.githubusercontent.com/u/43428393?v=4
https://api.github.com/users/zeyuyun1/events{/privacy}
https://api.github.com/users/zeyuyun1/followers
https://api.github.com/users/zeyuyun1/following{/other_user}
https://api.github.com/users/zeyuyun1/gists{/gist_id}
https://github.com/zeyuyun1
zeyuyun1
MDQ6VXNlcjQzNDI4Mzkz
https://api.github.com/users/zeyuyun1/orgs
https://api.github.com/users/zeyuyun1/received_events
https://api.github.com/users/zeyuyun1/repos
https://api.github.com/users/zeyuyun1/starred{/owner}{/repo}
https://api.github.com/users/zeyuyun1/subscriptions
User
https://api.github.com/users/zeyuyun1
public
MEMBER
Yes it shouldn’t fail with older version of transformers since this is only a special feature to make caching more efficient when using transformers for tokenization.
We’ll update this.
2020-09-10T06:25:21Z
https://github.com/huggingface/datasets/issues/598#issuecomment-690017255
https://api.github.com/repos/huggingface/datasets/issues/598
MDEyOklzc3VlQ29tbWVudDY5MDAxNzI1NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/690017255/reactions
2020-09-10T06:25:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/690017255
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
Instead of downloading the package from pip, downloading the version from source will result in an error when loading dataset (the pip version is completely fine):

To recreate the error: 
First, installing nlp directly from source:
```
git clone https://github.com/huggingface/nlp.git
cd nlp
pip install -e .
```
Then run:
```
from nlp import load_dataset
dataset = load_dataset('wikitext', 'wikitext-2-v1',split = 'train') 
```
will give error:

```
>>> dataset = load_dataset('wikitext', 'wikitext-2-v1',split = 'train')
Checking /home/zeyuy/.cache/huggingface/datasets/84a754b488511b109e2904672d809c041008416ae74e38f9ee0c80a8dffa1383.2e21f48d63b5572d19c97e441fbb802257cf6a4c03fbc5ed8fae3d2c2273f59e.py for additional imports.
Found main folder for dataset https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/wikitext.py at /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext
Found specific version folder for dataset https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/wikitext.py at /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d
Found script file from https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/wikitext.py to /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d/wikitext.py
Found dataset infos file from https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/dataset_infos.json to /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d/dataset_infos.json
Found metadata file for dataset https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/wikitext.py at /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d/wikitext.json
Loading Dataset Infos from /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d
Overwrite dataset info from restored data version.
Loading Dataset info from /home/zeyuy/.cache/huggingface/datasets/wikitext/wikitext-2-v1/1.0.0/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d
Reusing dataset wikitext (/home/zeyuy/.cache/huggingface/datasets/wikitext/wikitext-2-v1/1.0.0/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d)
Constructing Dataset for split train, from /home/zeyuy/.cache/huggingface/datasets/wikitext/wikitext-2-v1/1.0.0/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/load.py", line 600, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/builder.py", line 611, in as_dataset
    datasets = utils.map_nested(
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/utils/py_utils.py", line 216, in map_nested
    return function(data_struct)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/builder.py", line 631, in _build_single_dataset
    ds = self._as_dataset(
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/builder.py", line 704, in _as_dataset
    return Dataset(**dataset_kwargs)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/arrow_dataset.py", line 188, in __init__
    self._fingerprint = generate_fingerprint(self)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/fingerprint.py", line 91, in generate_fingerprint
    hasher.update(key)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/fingerprint.py", line 57, in update
    self.m.update(self.hash(value).encode("utf-8"))
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/fingerprint.py", line 53, in hash
    return cls.hash_default(value)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/fingerprint.py", line 46, in hash_default
    return cls.hash_bytes(dumps(value))
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/utils/py_utils.py", line 361, in dumps
    with _no_cache_fields(obj):
  File "/home/zeyuy/miniconda3/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/utils/py_utils.py", line 348, in _no_cache_fields
    if isinstance(obj, tr.PreTrainedTokenizerBase) and hasattr(obj, "cache") and isinstance(obj.cache, dict):
AttributeError: module 'transformers' has no attribute 'PreTrainedTokenizerBase'

```


