Back
+20 XP
Concept

Lexical Analysis & Tokenizing with `tokenize`

Lexing decomposes raw source strings into semantic tokens: identifiers, numbers, operators, and keywords. Python's built-in `tokenize` module converts UTF-8 byte streams into structured `TokenInfo` tuples 🔍.

python
import io
import tokenize

code = b"total = 42 + price"
stream = io.BytesIO(code)
for t in tokenize.tokenize(stream.readline):
    if t.type not in (tokenize.ENCODING, tokenize.ENDMARKER):
        print(f"{tokenize.tok_name[t.type]}: {t.string!r} 🔍")

# 61. Lexical Analysis & Tokenizing with `tokenize` Lexing decomposes raw source strings into semantic tokens: identifiers, numbers, operators, and keywords. Python's built-in `tokenize` module converts UTF-8 byte streams into structured `TokenInfo` tuples 🔍. ```python import io import tokenize code = b"total = 42 + price" stream = io.BytesIO(code) for t in tokenize.tokenize(stream.readline): if t.type not in (tokenize.ENCODING, tokenize.ENDMARKER): print(f"{tokenize.tok_name[t.type]}: {t.string!r} 🔍") ``` The lexer tracks line and column numbers for every token. When Python raises a `SyntaxError`, the parser uses these token coordinates to point the exact carrot indicator `^` at your typo 🎯! --- ### Question What is the primary role of a lexical analyzer (tokenizer) in a compilation pipeline? 🧠 - [ ] a. Converting unstructured character streams into categorized atomic tokens while filtering noise 🔍 - [ ] b. Compiling bytecode directly into native AMD64 CPU instructions - [ ] c. Allocating heap memory chunks for active thread stack frames - [ ] d. Executing arithmetic expressions and piping output to the kernel --- ### Challenge Use `tokenize.tokenize` on `b'a + 10'`. Extract and collect all `NUMBER` token strings (`t.string`). Print the resulting list. 🎯 <details> <summary>Solution</summary> ```python import io import tokenize code = b'a + 10' numbers = [ t.string for t in tokenize.tokenize(io.BytesIO(code).readline) if t.type == tokenize.NUMBER ] print(numbers) ``` </details>

Checkpoint

What is the primary role of a lexical analyzer (tokenizer) in a compilation pipeline? 🧠

Practice

Use `tokenize.tokenize` on `b'a + 10'`. Extract and collect all `NUMBER` token strings (`t.string`). Print the resulting list. 🎯

Terminal
01
Tap code blocks below to assemble
Available Blocks