โ† Compiler Design learning path
Compiler Design ยท Module 2 of 10

Lexical analysis

Turning characters into tokens.

๐Ÿ“– Key ideas

  1. The lexer (scanner) reads characters and groups them into tokens: keywords, identifiers, numbers, operators.
  2. A lexeme is the actual text; a token is its category; a pattern describes which lexemes form a token.
  3. Tokens are specified with regular expressions and recognised with finite automata.
  4. The lexer removes whitespace and comments and enters identifiers in the symbol table.
  5. Tools: Lex and Flex generate lexers from regular expression rules.
๐Ÿ’ก Real-life example: Like reading a sentence and marking each word as noun, verb or punctuation before checking grammar.
TokenCategory of a lexeme, such as identifier
LexemeActual characters matched
PatternRule describing lexemes of a token
Lex/FlexLexer generator tools