Token Stream Modification with tokenize.untokenize
While ASTs discard comments and whitespace, token streams retain every layout detail. `tokenize.untokenize` reconstructs valid source code directly from modified token streams, making it ideal for non-destructive code rewriting 🖋️.
import io
import tokenize
src = b"count = 10 # Initial seed\n"
tokens = list(tokenize.tokenize(io.BytesIO(src).readline))
# Mutate token string in-place:
mutated = [
t._replace(string="99") if (t.type == tokenize.NUMBER and t.string == "10") else t
for t in tokens
]
output = tokenize.untokenize(mutated).decode("utf-8")
print(output.strip())
# 71. Token Stream Modification with tokenize.untokenize While ASTs discard comments and whitespace, token streams retain every layout detail. `tokenize.untokenize` reconstructs valid source code directly from modified token streams, making it ideal for non-destructive code rewriting 🖋️. ```python import io import tokenize src = b"count = 10 # Initial seed\n" tokens = list(tokenize.tokenize(io.BytesIO(src).readline)) # Mutate token string in-place: mutated = [ t._replace(string="99") if (t.type == tokenize.NUMBER and t.string == "10") else t for t in tokens ] output = tokenize.untokenize(mutated).decode("utf-8") print(output.strip()) ``` Because AST parsing (`ast.parse`) throws away all comment tokens and spacing nuances, tools like LibCST and Bowler rely on token-level trees to perform formatting-safe codebase migrations 🚀! --- ### Question Why do automated refactoring formatters operate on token streams or CSTs rather than standard ASTs? 🧠 - [ ] a. Token streams preserve inline comments, exact whitespace formatting, and indentation that ASTs strip 🖋️ - [ ] b. Token streams execute directly inside the CPU register cache - [ ] c. AST trees cannot parse complex binary operators - [ ] d. Token streams automatically generate binary C headers --- ### Challenge Use `tokenize.tokenize` on `b'score = 50\n'`. Replace the NUMBER token `'50'` with `'100'`. Untokenize with `tokenize.untokenize` and print the decoded string stripped. 🎯 <details> <summary>Solution</summary> ```python import io import tokenize code = b'score = 50\n' tokens = list(tokenize.tokenize(io.BytesIO(code).readline)) mod = [ t._replace(string='100') if (t.type == tokenize.NUMBER and t.string == '50') else t for t in tokens ] out = tokenize.untokenize(mod).decode('utf-8').strip() print(out) ``` </details>