Why Transformers Replaced RNNs

Please join my mailing list here 👉 https://briankeating.com/list  to win a meteorite 💥

David Sussillo spent years training neural networks at Google Brain

and now leads research at Meta Reality Labs. His verdict on the

path to artificial consciousness might surprise you.

In this conversation, we discuss why scaling transformers alone

won't produce conscious AI, how recurrent neural networks differ

from the transformer architectures powering today's LLMs, the

FORCE algorithm he developed to train RNNs, what his work on

Meta's EMG wristband reveals about AI and biological signals,

Rich Sutton's Bitter Lesson and whether we're locked into a

GPU-transformer paradigm, and why studying the brain may be

the only way out of AI's current limitations.

David is also the author of Emergence, a braided memoir that

tells the story of his path from foster care to the frontier

of neuroscience and AI research.

David Sussillo is a research scientist at Meta Reality Labs

and an adjunct professor at Stanford University. He completed

his PhD at Columbia University under Larry Abbott and previously

worked at Google Brain.

Key Takeaways:

00:00 Scale alone won't produce consciousness — here's why 

00:25 The "bitter lesson": big data crushes every algorithmic invention 

01:15 How hallucinations were discovered early at Google 

05:00 The lock-in problem: why LLM + GPU may cap AI's ceiling 

08:25 What Meta's neural wristband is actually trying to solve 

11:20 Rich Sutton's bitter lesson explained 

14:15 "More is Different" — why you can't reduce complexity upward 

28:30 The book Emergence — two stories braided into one 

36:00 Why mentorship matters more than intelligence for at-risk kids 

47:00 Stability over intensity: what kids in foster care actually need

 ➡️ Follow David Sussillo 

🌐 Website / Research: Meta Reality Labs 

📚 Emergence (memoir): https://www.amazon.com/Emergence-Memoir-Boyhood-Computation-Mysteries/dp/1538768577 

🎓 Stanford / Columbia / Carnegie Mellon Alumni

✍️ Email: davidsussillo@gmail.com 

🏄‍♂️ Twitter: https://x.com/SussilloDavid

Join this channel to get access to perks like monthly Office Hours:

https://www.youtube.com/channel/UCmXH_moPhfkqCk6S3b9RWuw/join

📚 Get my books:

Think Like a Nobel Prize Winner, with productivity tips from 9 Nobel Prize winners: https://a.co/d/03ezQFu

Focus Like a Nobel Prize Winner, with life-changing interviews with 9 Nobel Prizewinners: https://a.co/d/hi50U9U

My tell-all cosmic memoir Losing the Nobel Prize: http://amzn.to/2sa5UpA

The first-ever audiobook from Galileo: Dialogue Concerning the Two Chief World Systems: Ptolemaic and Copernican https://a.co/d/iZPi9Un

Follow me to ask questions of my guests:

🏄‍♂️ Twitter: https://twitter.com/DrBrianKeating

🔔 Subscribe https://www.youtube.com/DrBrianKeating?sub_confirmation=1

📝 Join my mailing list; just click here http://briankeating.com/list

✍️ Detailed Blog posts here: https://briankeating.com/blog

🎙️ Listen on audio-only platforms: https://briankeating.com/podcast

#universe #podcast #briankeating #intotheimpossible #science #astronomy #cosmology #cosmicmicrowavebackground #intotheimpossible #briankeating

Learn more about your ad choices. Visit megaphone.fm/adchoices