
<li class="listitem">
                to optionally assign a token id to a token type,
              </li>
<li class="listitem">
                to optionally associate some code to execute whenever an instance
                of this token type has been matched,
              </li>
<li class="listitem">
                and to optionally specify the attribute type of the token value.
              </li>
</ul></div>
<p>
            The token itself is a data structure returned by the lexer iterators.
      