Sergey Levine - Building LLMs for the Physical World - [Invest Like the Best, EP.465]
Invest Like the Best with Patrick O'Shaughnessy
From End-to-End Control to Multimodal Common Sense
Sergey Levine traces robotics milestones from early end-to-end systems to deep reinforcement learning and multimodal models that supply common-sense knowledge.
My guest today is Sergey Levine, a professor at UC Berkeley and co-founder of Physical Intelligence. The company is building robotic foundation models designed to control any embodied system to do any task in any environment.
Sergey argues that solving robotics at full generality is the right path, and that building systems that learn across many robots, environments, and tasks may be the more scalable approach than building narrow specialists. We discuss how these models can perform new tasks without being trained on them directly, and why everyday human actions remain the hardest problems in the field.
He also reflects on how human trust and acceptance may matter as much as technical breakthroughs in determining when robots become part of daily life.
Please enjoy my conversation with Sergey Levine.
For the full show notes, transcript, and links to mentioned content, check out the episode page here.
-----
Become a Colossus member to get our quarterly print magazine and private audio experience, including exclusive profiles and early access to select episodes. Subscribe at colossus.com/subscribe.
-----
Ramp’s mission is to help companies manage their spend in a way that reduces expenses and frees up time for teams to work on more valuable projects. Go to ramp.com/invest to sign up for free and get a $250 welcome bonus.
-----
Trusted by thousands of businesses, Vanta continuously monitors your security posture and streamlines audits so you can win enterprise deals and build customer trust without the traditional overhead. Visit vanta.com/invest.
-----
WorkOS is a developer platform that enables SaaS companies to quickly add enterprise features to their applications. Visit WorkOS.com to transform your application into an enterprise-ready solution in minutes, not months.
-----
Rogo is the AI platform for finance. They're building agents for Wall Street that are trained to understand how bankers and investors actually do work: from diligence and modeling, to turning analysis into deliverables. To learn more, visit rogo.ai/invest.
-----
Ridgeline has built a complete, real-time, modern operating system for investment managers. It handles trading, portfolio management, compliance, customer reporting, and much more through an all-in-one real-time cloud platform. Visit ridgelineapps.com.
-----
Editing and post-production work for this episode was provided by The Podcast Consultant (https://thepodcastconsultant.com).
Timestamps:
(00:00:00) Welcome to Invest Like the Best
(00:02:43) Intro: Sergey Levine
(00:03:29) Why Bet on Generality Over Specialization
(00:07:24) What if PI succeeds?
(00:09:05) Pros and Cons of Humanoid Robotics
(00:11:02) Timeline of Major Milestones in Robotics
(00:15:47) Sergey's Personal Journey
(00:18:22) Making General Intelligence Happen
(00:19:57) Understanding Robot Data Collection
(00:22:12) Most Surprising Discovery at Physical Intelligence
(00:24:48) The Science of Common Sense
(00:25:36) Long-Range Tasks in Robotics
(00:27:24) Why Wouldn’t We Have A Robot in Our Kitchen by 2050
(00:31:21) Other Interesting Approaches
(00:32:38) Cool vs. Useful in Robotics
(00:36:48) Form Factor Innovation
(00:38:22) Physical Intelligence Analogy
(00:39:30) Economic Transformation from Robotics
(00:40:48) Controversies in the Robotics Community
(00:42:16) Arguments Against End-to-End Learning
(00:42:34) Compositional Learning Explained
(00:43:25) Last Tasks Robots will Conquer
(00:44:30) Dark Parts of the Robotics Brain
(00:47:05) What Makes a Great Researcher
(00:50:15) Manufacturing and Scale Challenges
(00:51:17) How Companies Should Prepare for Robotics
(00:53:38) Boston Dynamics' Demos
(00:55:43) Converging Technologies Enabling Robotics
(00:56:47) How to Stay Up To Date in Robotics
(00:59:51) Near Term Objectives
(01:00:49) Confidence Level Among Researchers
(01:03:31) Google's Experimentation Culture
(01:04:24) The Kindest Thing
The AI-powered Podcast Player
