 </span>
      
      <p class="shortcomment cmt">Tokenize considering special tokens and split algorithm</p><div class="fullcomment"><div class="comment cmt"><p>Tokenize considering special tokens and split algorithm</p></div><dl class="attributes block"> <dt>Definition Classes</dt><dd>BpeTokenizer</dd></dl></div>
    </li><li name="com.johnsnowlabs.nlp.annotators.tokenizer.bpe.Phi4Tokenizer#tokenizeSubText" visbl="pub" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped">
      <a id=