token definition is used to describe the main features of a particular
            token type, especially:
          </p>
<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
<li class="listitem">
                to simplify the definition of a token type using a regular expression
                pattern applied while matching this token type,
              </li>
<li class="listitem">
                to associate a token type with a particular lexer state,
              </li>