the word is split into smaller subwords, until they are known.
</p></div><dl class="paramcmts block"><dt>returns</dt><dd class="cmt"><p>
  Array of TokenPieces, corresponding to encoded token</p></dd></dl><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd>BpeTokenizer</dd></dl></div>
    </li><li name="com.johnsnowlabs.nlp.annotators.tokenizer.bpe.BpeTokenizer#bpeRanks" visbl="prt" class="indented0 " data-isabs="false" fullComment="yes" group="Ungrouped">
    