ubroutine and it is meant for the string passed to
the <em>first_token()</em> function. If you want to reuse it for a
different string, but the same definition, simply use
<em>first_token()</em> on the new string.

<dl>

<dt><a name="1"><b class='cmd'>use tokenize</b> </a><dd>

To import the definitions, use this module.

<br><br>
<dt><a name="2"><b class='cmd'>call set_tokenizer( token, gaps, separators, delimiters )</b> </a><dd>

Initialise the tokenizer &quot;token&quot; with various sets o