le>
<h2><a name="description">DESCRIPTION</a></h2>
<p>

The <em>tokenize</em> module provides a method to split a string
into parts according to some simple rules:
<ul>
<li>
A string can be split into &quot;words&quot; by considering spaces and commas and
such as separating the words. Two or more such characters are treated as
a single such separator. In the terminology of the module they represent
gaps of varying width. As a consequence there are no zero-length words.

<br><br>
<li>
A string