Tirohia te whiti i raro nei, ka whakautu i te pātai i muri mai i roto i te reo ano rite te whiti: Passage: Tokens e tautuhia pinepine e faaiteraa auau, e e mārama e te generator analyzer lexical pērā i lex. te analyzer lexical (whakaputa aunoa e te taputapu rite lex, ranei hanga-a-ringa) pānui i roto i te awa o pūāhua, tautuhi i te lexemes i roto i te awa, ka whakarōpū ratou ki tokens. huaina tenei tokenizing. ki te kitea e te lexer he tohu hē, ka pūrongo te reira i te hapa. pātai: He aha ka kitea e te reira, ki te pūrongo te reira i te hapa?
he tohu kore