โ Compiler Design learning pathCompiler Design ยท Module 2 of 10
Lexical analysis
Turning characters into tokens.
๐ Key ideas
- The lexer (scanner) reads characters and groups them into tokens: keywords, identifiers, numbers, operators.
- A lexeme is the actual text; a token is its category; a pattern describes which lexemes form a token.
- Tokens are specified with regular expressions and recognised with finite automata.
- The lexer removes whitespace and comments and enters identifiers in the symbol table.
- Tools: Lex and Flex generate lexers from regular expression rules.
๐ก Real-life example: Like reading a sentence and marking each word as noun, verb or punctuation before checking grammar.
TokenCategory of a lexeme, such as identifier
LexemeActual characters matched
PatternRule describing lexemes of a token
Lex/FlexLexer generator tools