Yaroslav Bulatov helped build the AI era from the inside, as one of the earliest researchers at both OpenAI and Google Brain. Now he wants to tear it all down and start over. Modern deep learning, he argues, is up to 100x more wasteful than it needs to be - a Frankenstein of hacks designed for the wrong hardware. With a power wall approaching in two years, Yaroslav is leading an open effort to reinvent AI from scratch: no backprop, no legacy assumptions, just the benefit of hindsight and AI agents that compress decades of research into months. Along the way, we dig into why AGI is a "religious question," how a sales guy with no ML background became one of his most productive contributors, and why the Muon optimizer, one of the biggest recent breakthroughs, could only have been discovered by a non-expert.
Timeline
00:12 — Introduction and Yaroslav's background at OpenAI and Google Brain
01:16 — Why deep learning isn't such a good idea
02:03 — The three definitions of AGI: religious, financial, and vibes-based
07:52 — The SAI framework: do we need the term AGI at all?
10:58 — What matters more than AGI: efficiency and refactoring the AI stack
13:28 — Jevons paradox and the coming energy wall
14:49 — The recipe: replaying 70 years of AI with hindsight
17:23 — Memory, energy, and gradient checkpointing
18:34 — Why you can't just optimize the current stack (the recurrent laryngeal nerve analogy)
21:05 — What a redesigned AI might look like: hierarchical message passing
22:31 — Can a small team replicate decades of research?
24:23 — Why non-experts outperform domain specialists
27:42 — The GPT-2 benchmark: what success looks like
29:01 — Ian Goodfellow, Theano, and the origins of TensorFlow
30:12 — The Muon optimizer origin story and beating Google on ImageNet
36:16 — AI coding agents for software engineering and research
40:12 — 10-year outlook and the voice-first workflow
42:23 — Why start with text over multimodality
45:13 — Are AI labs like SSI on the right track?
48:52 — Getting rid of backprop — and maybe math itself
53:57 — The state of ML academia and NeurIPS culture
56:41 — The Sutra group challenge: inventing better learning algorithms
Music:
- "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
- "Palms Down" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
- Changes: trimmed
About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.
