core.py Episode 7: The Old Parser
7 snips
Jan 31, 2024 Delve into the quirks of Python's tokenizer, from invisible tokens to the historical backticks that shaped syntax. Explore the evolution of Python grammar and parsing techniques, emphasizing the challenges of ambiguity and efficiency. Discover the transition from Python 2 to 3 and the role of lib223 in maintaining code integrity. Laugh at the quirks of backslashes and context managers, and learn about recent enhancements in error handling and memory performance in CPython. It's a wild ride through Python's parsing journey!
AI Snips
Chapters
Transcript
Episode notes
Avoid Unnecessary Backslashes
- Backslashes in Python source code break the visual indentation paradigm and hinder parsing and formatting.
- Where possible, remove unnecessary backslashes to improve code clarity and formatability.
With Statement Can't Use LL1
- LL1 parsers cannot resolve grammar for 'with' statements with multiple context managers and parentheses due to lookahead limits.
- This limitation forced reliance on complex solutions and revealed the need for an LR parser to handle such syntax.
New Exception for Incomplete Input
- Python REPL needs to distinguish incomplete input from syntax errors to allow multi-line code entry.
- A new internal exception 'IncompleteInputError' replaces unreliable heuristics to better handle partial input.