pan class="extype" name="scala.Array">Array</span>[<a href="../../common/IndexedToken.html" class="extype" name="com.johnsnowlabs.nlp.annotators.common.IndexedToken">IndexedToken</a>]</span>
      </span>
      
      <p class="shortcomment cmt">Tokenize considering special tokens and split algorithm</p><div class="fullcomment"><div class="comment cmt"><p>Tokenize considering special tokens and split algorithm</p></div><dl class="attributes block"> <dt>Definition Classes</dt><dd>BpeTokenizer</dd></dl></div>