Best practices for training tokenizers with nlp
COLLABORATOR
Docs that explain how to train a tokenizer with `datasets` are available here: https://huggingface.co/docs/tokenizers/training_from_memory#using-the-datasets-library
2022-10-04T17:28:04Z
https://github.com/huggingface/datasets/issues/541#issuecomment-1267327660
https://api.github.com/repos/huggingface/datasets/issues/541
IC_kwDODunzps5LieKs
https://api.github.com/repos/huggingface/datasets/issues/comments/1267327660/reactions
2022-10-04T17:28:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1267327660
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
Hi, thank you for developing this library. 

What do you think are the best practices for training tokenizers using `nlp`?  In the document and examples, I could only find pre-trained tokenizers used.