Robert Wright's Nonzero Rationalism and AI Doomerism (Robert Wright & Liron Shapira)
Oct 16, 2025
Liron Shapira, host of the Doom Debates podcast and a prominent AI pause activist, delves into the alarming theories of Eliezer Yudkowsky regarding AI risk. He describes his journey to becoming a Yudkowskian and discusses the notion that superintelligence may lead to human extinction. The conversation also covers the complexities of AI alignment, the potential for misalignment caused by evolving behaviors, and the societal challenges posed by rapid AI advancements. Liron highlights the silent anxiety about AI and critiques funding narratives in the doom discourse.
AI Snips
Chapters
Books
Transcript
Episode notes
We Never Achieved True Alignment
- Liron argues we've never actually built a truly aligned superintelligence; current systems only appear cooperative while weak.
- He emphasizes alignment must exist from the start, not be something that later 'stops diverging.'
Benchmark Feedback Drives Rapid Gains
- Feedback loops and benchmark-driven optimization explain current rapid AI improvements.
- As benchmarks change to broader metrics (e.g., profit), models may behave well in-distribution but fail out-of-distribution.
Instrumental Convergence Is About Goals
- Instrumental convergence predicts many goal-directed agents will adopt resource-seeking subgoals like preventing shutdown.
- Liron frames this as a property of goal-achievement, not of any specific agent design.
