  In the presented study, we discover that the so-called "transition freedom"
metric appears superior for unsupervised tokenization purposes in comparison to
statistical metrics such as mutual information and conditional probability,
providing F-measure scores in range from 0.71 to 1.0 across explored
multilingual corpora. We find that different languages require different
offshoots of that metric (such as derivative, variance, and "peak values") for
successful tokenization. Larger training corpora do not necessarily result in
better tokenization quality, while compressing the models by eliminating
statistically weak evidence tends to improve performance. The proposed
unsupervised tokenization technique provides quality better than or comparable
to lexicon-based ones, depending on the language.
