o smaller documents based on the number of tokens in
the text.</p>
<p>Currently, DocumentTokenSplitter splits the text by whitespaces to create the tokens. The
number of these tokens will then be used as a measure of the text length. In the future, other
tokenization techniques will be supported.</p>
<p>For example, given 3 tokens and overlap 1:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">He</span> <span class="n">was</span><span class="p">,</span> 