"><b>Figure&#160;8.&#160;The structure of a token</b></p>
<div class="figure-contents"><span class="inlinemediaobject"><img src="../../../.././images/tokenstructure.png" alt="The structure of a token"></span></div>
</div>
<p><br class="figure-break">
          </p>
<p>
            The token value and the lexer state the token has been recognized in
            may be omitted for optimization reasons, thus avoiding the need for the
            token to carry more data than actually required. This configurati