Combining LLMs with Robot Learning

Karol explains experiments marrying internet-scale knowledge from LLMs with robot motion models to transfer prior knowledge.

Play episode from 26:00

chevron_right

Transcript

chevron_right

Transcript

Episode notes

Karol Hausman is the co-founder and CEO of Physical Intelligence, a robotics company building a general-purpose “AI brain for the physical world.” The company has raised more than $1 billion in funding to develop foundation models that allow robots to operate across many machines, environments, and tasks rather than being programmed for a single purpose. The core thesis: the same scaling dynamics that transformed language models may also unlock robotic intelligence. But only if you resist every commercial pressure pushing you toward specialization. The central challenge isn’t mechanical design. It’s intelligence: how robots learn, generalize, and interact with a physical world that is far harder to simulate than it is to describe. Before launching Physical Intelligence, Karol worked at Google Brain and Stanford University, studying robot learning alongside researchers Sergey Levine and Chelsea Finn, who later became his co-founders.

In our conversation, we explore:

How growing up in a small town in Poland and watching Star Wars sparked Karol’s fascination with robots
The moment a lecture from Sergey Levine convinced him to abandon his PhD research direction and pivot fully to deep learning
Why robotics has historically lagged behind breakthroughs in language models
The case for building a general “AI brain” for the physical world rather than a single specialized robot
The role of real-world data in training robots, the limits of simulation, and how deployment could create a powerful data flywheel
The return of reinforcement learning and the parallels between human learning and robot training
The unique challenges of physical intelligence and why robots must operate with far higher reliability than language models

—

Thank you to the partners who make this possible

Brex: The intelligent finance platform.

Granola: The app that might actually make you love meetings.

—

Transcript: https://www.generalist.com/p/karol-hausman-physical-intelligence

—

Timestamps

(00:00) Intro

(04:05) Karol’s early fascination with robots

(07:38) How Karol relates to Fei-Fei Li’s biography

(08:52) What inspired Karol to build better robots