Invest Like the Best with Patrick O'Shaughnessy

Sergey Levine - Building LLMs for the Physical World - [Invest Like the Best, EP.465]

147 snips
Mar 31, 2026
Guest
Sergey Levine
Sergey Levine, UC Berkeley professor and robotics AI researcher, explores building foundation models for the physical world. He digs into why general-purpose robots may beat narrow specialists. They discuss dexterity, common sense, language-guided tasks, and why homes are the toughest frontier. There is also a look at trust, cheap hardware, and robots augmenting work before replacing it.
Ask episode
AI Snips
Chapters
Books
Transcript
Episode notes
INSIGHT

Multimodal LLMs Offer A Path To Robot Common Sense

  • Robotics long lacked a source of common sense for rare edge cases, even when the robot could perform the core task.
  • Sergey Levine says multimodal LLMs now provide a path because they contain broad world knowledge, though grounding that knowledge into a robot remains hard.
ANECDOTE

Sergey Levine's Path From Graphics To Robot Learning

  • Sergey Levine moved from computer graphics into robotics to build systems that improve through experience rather than fixed programming.
  • At Google he scaled learning across many robots, then concluded robotics needs both web-scale prior knowledge and reinforcement learning, not either alone.
INSIGHT

How Physical Intelligence Combines LLMs And RL

  • Sergey Levine's recipe starts with a vision-language-action model, then adds chain-of-thought reasoning for edge cases and reinforcement learning for speed and robustness.
  • In espresso-making, repeated practice improved throughput after the model first absorbed web knowledge and diverse robot data.
Get the Snipd Podcast app to discover more snips from this episode
Get the app