ource code for omterms.tokenizer</h1><div class="highlight"><pre>
<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="sd">&quot;&quot;&quot;OpenMaker text tokenizer.</span>

<span class="sd">Author: Bulent Ozel</span>
<span class="sd">e-mail: bulent.ozel@gmail.com</span>

<span class="sd">The module contains a set of basic tools in order to tokenize a given inout text.</span>

<span class="sd">Todo:</span>
<span class="sd">    * Nothing at the moment ;)</span>

<span class="sd">&quot;&