What tokenizer is best?
#2
by turian - opened
What tokenizer is best to use with flair/pos-english?
It was not clear to me from the model card what tokenizer was used during training. And the default FLAIR tokenizer is relatively simple.
The tagger was trained on Ontonotes, so it reuses the tokenization from this dataset.