WikiText 103 tokenizer hangs
NONE
any updates on this? It happens to me on [OpenWikiText-20%](https://huggingface.co/datasets/Bingsu/openwebtext_20p) dataset, but not on [OpenWebText-10k](https://huggingface.co/datasets/stas/openwebtext-10k). This is really strange because I don't change anything else in my running script.

transformers version 4.18.0.dev0
datasets version 1.18.0
2023-08-08T08:40:30Z
https://github.com/huggingface/datasets/issues/5131#issuecomment-1669178300
https://api.github.com/repos/huggingface/datasets/issues/5131
IC_kwDODunzps5jfaO8
https://api.github.com/repos/huggingface/datasets/issues/comments/1669178300/reactions
2023-08-08T08:42:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1669178300
https://avatars.githubusercontent.com/u/50806659?v=4
https://api.github.com/users/ionutmodo/events{/privacy}
https://api.github.com/users/ionutmodo/followers
https://api.github.com/users/ionutmodo/following{/other_user}
https://api.github.com/users/ionutmodo/gists{/gist_id}
https://github.com/ionutmodo
ionutmodo
MDQ6VXNlcjUwODA2NjU5
https://api.github.com/users/ionutmodo/orgs
https://api.github.com/users/ionutmodo/received_events
https://api.github.com/users/ionutmodo/repos
https://api.github.com/users/ionutmodo/starred{/owner}{/repo}
https://api.github.com/users/ionutmodo/subscriptions
User
https://api.github.com/users/ionutmodo
public
See issue here: https://github.com/huggingface/transformers/issues/19702 