Bookcorpus data contains pretokenized text
MEMBER
Yes indeed it looks like some `'` and spaces are missing (for example in `dont` or `didnt`).
Do you know if there exist some copies without this issue ?
How would you fix this issue on the current data exactly ? I can see that the data is raw text (not tokenized) so I'm not sure I understand how you would do it. Could you provide more details ?
2020-08-31T15:52:02Z
https://github.com/huggingface/datasets/issues/486#issuecomment-683865629
https://api.github.com/repos/huggingface/datasets/issues/486
MDEyOklzc3VlQ29tbWVudDY4Mzg2NTYyOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/683865629/reactions
2020-08-31T15:52:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/683865629
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I'm afraid that I don't know how to obtain the original BookCorpus data. I believe this version came from an anonymous Google Drive link posted in another issue.

Going through the raw text in this version, it's apparent that NLTK's TreebankWordTokenizer was applied on it (I gave some examples in my original post), followed by:
`' '.join(tokens)`
You can retrieve the tokenization by splitting on whitespace. You can then "detokenize" it with TreebankWordDetokenizer class of NLTK (though, as I suggested, use the fixed version in my repo). This will bring the text closer to its original form, but some steps of TreebankWordTokenizer are destructive, so it wouldn't be one-to-one. Something along the lines of the following should work:
```
treebank_detokenizer = nltk.tokenize.treebank.TreebankWordDetokenizer()
db = nlp.load_dataset('bookcorpus', split=nlp.Split.TRAIN)
db = db.map(lambda x: treebank_detokenizer.detokenize(x['text'].split()))
```

Regarding other issues beyond the above, I'm afraid that I can't help with that.
2020-09-01T13:11:19Z
https://github.com/huggingface/datasets/issues/486#issuecomment-684841693
https://api.github.com/repos/huggingface/datasets/issues/486
MDEyOklzc3VlQ29tbWVudDY4NDg0MTY5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/684841693/reactions
2020-09-01T13:11:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684841693
https://avatars.githubusercontent.com/u/99543?v=4
https://api.github.com/users/orsharir/events{/privacy}
https://api.github.com/users/orsharir/followers
https://api.github.com/users/orsharir/following{/other_user}
https://api.github.com/users/orsharir/gists{/gist_id}
https://github.com/orsharir
orsharir
MDQ6VXNlcjk5NTQz
https://api.github.com/users/orsharir/orgs
https://api.github.com/users/orsharir/received_events
https://api.github.com/users/orsharir/repos
https://api.github.com/users/orsharir/starred{/owner}{/repo}
https://api.github.com/users/orsharir/subscriptions
User
https://api.github.com/users/orsharir
public
MEMBER
Ok I get it, that would be very cool indeed

What kinds of patterns the detokenizer can't retrieve ?
2020-09-08T09:36:00Z
https://github.com/huggingface/datasets/issues/486#issuecomment-688748088
https://api.github.com/repos/huggingface/datasets/issues/486
MDEyOklzc3VlQ29tbWVudDY4ODc0ODA4OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/688748088/reactions
2020-09-08T09:36:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688748088
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
The TreebankTokenizer makes some assumptions about whitespace, parentheses, quotation marks, etc. For instance, while tokenizing the following text:
```
Dwayne "The Rock" Johnson
```
will result in:
```
Dwayne `` The Rock '' Johnson
```
where the left and right quotation marks are turned into distinct symbols. Upon reconstruction, we can attach the left part to its token on the right, and respectively for the right part. However, the following texts would be tokenized exactly the same:
```
Dwayne " The Rock " Johnson
Dwayne " The Rock" Johnson
Dwayne     " The Rock" Johnson
...
```
In the above examples, the detokenizer would correct these inputs into the canonical text
```
Dwayne "The Rock" Johnson
```
However, there are cases where there the solution cannot easily be inferred (at least without a true LM - this tokenizer is just a bunch of regexes). For instance, in cases where you have a fragment that contains the end of quote, but not its beginning, plus an accidental space:
```
... and it sounds fantastic, " he said.
```
In the above case, the tokenizer would assume that the quotes refer to the next token, and so upon detokenization it will result in the following mistake:
```
... and it sounds fantastic, "he said.
```

While these are all odd edge cases (the basic assumptions do make sense), in noisy data they can occur, which is why I mentioned that the detokenizer cannot restore the original perfectly.

2020-09-08T14:18:31Z
https://github.com/huggingface/datasets/issues/486#issuecomment-688908103
https://api.github.com/repos/huggingface/datasets/issues/486
MDEyOklzc3VlQ29tbWVudDY4ODkwODEwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/688908103/reactions
2020-09-08T14:18:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688908103
https://avatars.githubusercontent.com/u/99543?v=4
https://api.github.com/users/orsharir/events{/privacy}
https://api.github.com/users/orsharir/followers
https://api.github.com/users/orsharir/following{/other_user}
https://api.github.com/users/orsharir/gists{/gist_id}
https://github.com/orsharir
orsharir
MDQ6VXNlcjk5NTQz
https://api.github.com/users/orsharir/orgs
https://api.github.com/users/orsharir/received_events
https://api.github.com/users/orsharir/repos
https://api.github.com/users/orsharir/starred{/owner}{/repo}
https://api.github.com/users/orsharir/subscriptions
User
https://api.github.com/users/orsharir
public
NONE
To confirm, since this is preprocessed, this was not the exact version of the Book Corpus used to actually train the models described here (particularly Distilbert)?  https://huggingface.co/datasets/bookcorpus

Or does this preprocessing exactly match that of the papers?
2020-09-12T16:33:04Z
https://github.com/huggingface/datasets/issues/486#issuecomment-691513458
https://api.github.com/repos/huggingface/datasets/issues/486
MDEyOklzc3VlQ29tbWVudDY5MTUxMzQ1OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/691513458/reactions
2020-09-12T16:33:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/691513458
https://avatars.githubusercontent.com/u/1606458?v=4
https://api.github.com/users/arvieFrydenlund/events{/privacy}
https://api.github.com/users/arvieFrydenlund/followers
https://api.github.com/users/arvieFrydenlund/following{/other_user}
https://api.github.com/users/arvieFrydenlund/gists{/gist_id}
https://github.com/arvieFrydenlund
arvieFrydenlund
MDQ6VXNlcjE2MDY0NTg=
https://api.github.com/users/arvieFrydenlund/orgs
https://api.github.com/users/arvieFrydenlund/received_events
https://api.github.com/users/arvieFrydenlund/repos
https://api.github.com/users/arvieFrydenlund/starred{/owner}{/repo}
https://api.github.com/users/arvieFrydenlund/subscriptions
User
https://api.github.com/users/arvieFrydenlund
public
CONTRIBUTOR
I believe these are just artifacts of this particular source. It might be better to crawl it again, or use another preprocessed source, as found here: https://github.com/soskek/bookcorpus 
2020-09-20T09:11:31Z
https://github.com/huggingface/datasets/issues/486#issuecomment-695764048
https://api.github.com/repos/huggingface/datasets/issues/486
MDEyOklzc3VlQ29tbWVudDY5NTc2NDA0OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/695764048/reactions
2020-09-20T09:11:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/695764048
https://avatars.githubusercontent.com/u/99543?v=4
https://api.github.com/users/orsharir/events{/privacy}
https://api.github.com/users/orsharir/followers
https://api.github.com/users/orsharir/following{/other_user}
https://api.github.com/users/orsharir/gists{/gist_id}
https://github.com/orsharir
orsharir
MDQ6VXNlcjk5NTQz
https://api.github.com/users/orsharir/orgs
https://api.github.com/users/orsharir/received_events
https://api.github.com/users/orsharir/repos
https://api.github.com/users/orsharir/starred{/owner}{/repo}
https://api.github.com/users/orsharir/subscriptions
User
https://api.github.com/users/orsharir
public
CONTRIBUTOR
Yes actually the BookCorpus on hugginface is based on [this](https://github.com/soskek/bookcorpus/issues/24#issuecomment-643933352). And I kind of regret naming it as "BookCorpus" instead of something like "BookCorpusLike".

But there is a good news ! @shawwn has replicated BookCorpus in his way, and also provided a link to download the plain text files. see [here](https://github.com/soskek/bookcorpus/issues/27). There is chance we can have a "OpenBookCorpus" !
2020-09-30T11:58:11Z
https://github.com/huggingface/datasets/issues/486#issuecomment-701343912
https://api.github.com/repos/huggingface/datasets/issues/486
MDEyOklzc3VlQ29tbWVudDcwMTM0MzkxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/701343912/reactions
2020-09-30T12:01:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/701343912
https://avatars.githubusercontent.com/u/17963619?v=4
https://api.github.com/users/richarddwang/events{/privacy}
https://api.github.com/users/richarddwang/followers
https://api.github.com/users/richarddwang/following{/other_user}
https://api.github.com/users/richarddwang/gists{/gist_id}
https://github.com/richarddwang
richarddwang
MDQ6VXNlcjE3OTYzNjE5
https://api.github.com/users/richarddwang/orgs
https://api.github.com/users/richarddwang/received_events
https://api.github.com/users/richarddwang/repos
https://api.github.com/users/richarddwang/starred{/owner}{/repo}
https://api.github.com/users/richarddwang/subscriptions
User
https://api.github.com/users/richarddwang
public
COLLABORATOR
Resolved via #856
2022-10-04T17:44:29Z
https://github.com/huggingface/datasets/issues/486#issuecomment-1267345613
https://api.github.com/repos/huggingface/datasets/issues/486
IC_kwDODunzps5LiijN
https://api.github.com/repos/huggingface/datasets/issues/comments/1267345613/reactions
2022-10-04T17:44:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1267345613
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
It seem that the bookcoprus data downloaded through the library was pretokenized with NLTK's Treebank tokenizer, which changes the text in incompatible ways to how, for instance, BERT's wordpiece tokenizer works. For example, "didn't" becomes "did" + "n't", and double quotes are changed to `` and '' for start and end quotes, respectively.

On my own projects, I just run the data through NLTK's TreebankWordDetokenizer to reverse the tokenization (as best as possible). I think it would be beneficial to apply this transformation directly on your remote cached copy of the dataset. If you choose to do so, I would also suggest to use my fork of NLTK that fixes several bugs in their detokenizer (I've opened a pull-request, but they've yet to respond): https://github.com/nltk/nltk/pull/2575