:</span> <span class="s2">&quot; &quot;</span>
<span class="p">}</span>

<div class="viewcode-block" id="tokenize"><a class="viewcode-back" href="../../omterms.html#omterms.tokenizer.tokenize">[docs]</a><span class="k">def</span> <span class="nf">tokenize</span><span class="p">(</span><span class="n">raw</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;The function tokenizes by splitting them on spaces, line breaks or characters</span>
<span class="sd">        in CHARACTERS_TO_SPLIT.</