Memory, gradient checkpointing, and hardware mismatch

Yaroslav Bulatov helped build the AI era from the inside, as one of the earliest researchers at both OpenAI and Google Brain. Now he wants to tear it all down and start over. Modern deep learning, he argues, is up to 100x more wasteful than it needs to be  -  a Frankenstein of hacks designed for the wrong hardware. With a power wall approaching in two years, Yaroslav is leading an open effort to reinvent AI from scratch: no backprop, no legacy assumptions, just the benefit of hindsight and AI agents that compress decades of research into months. Along the way, we dig into why AGI is a "religious question," how a sales guy with no ML background became one of his most productive contributors, and why the Muon optimizer, one of the biggest recent breakthroughs, could only have been discovered by a non-expert.

Timeline

00:12 — Introduction and Yaroslav's background at OpenAI and Google Brain

01:16 — Why deep learning isn't such a good idea

02:03 — The three definitions of AGI: religious, financial, and vibes-based

07:52 — The SAI framework: do we need the term AGI at all?

10:58 — What matters more than AGI: efficiency and refactoring the AI stack

13:28 — Jevons paradox and the coming energy wall

14:49 — The recipe: replaying 70 years of AI with hindsight

17:23 — Memory, energy, and gradient checkpointing

18:34 — Why you can't just optimize the current stack (the recurrent laryngeal nerve analogy)

21:05 — What a redesigned AI might look like: hierarchical message passing

22:31 — Can a small team replicate decades of research?

24:23 — Why non-experts outperform domain specialists

27:42 — The GPT-2 benchmark: what success looks like

29:01 — Ian Goodfellow, Theano, and the origins of TensorFlow

30:12 — The Muon optimizer origin story and beating Google on ImageNet

36:16 — AI coding agents for software engineering and research

40:12 — 10-year outlook and the voice-first workflow

42:23 — Why start with text over multimodality

45:13 — Are AI labs like SSI on the right track?

48:52 — Getting rid of backprop — and maybe math itself

53:57 — The state of ML academia and NeurIPS culture

56:41 — The Sutra group challenge: inventing better learning algorithms

Music:

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
• "Palms Down" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
• Changes: trimmed

About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.