Agent reliability and the 100% Codex experiment

Sherwin Wu leads engineering for OpenAI’s API platform, where roughly 95% of engineers use Codex, often working with fleets of 10 to 20 parallel AI agents.

We discuss:

1. What OpenAI did to cut code review times from 10-15 minutes to 2-3 minutes

2. How AI is changing the role of managers

3. Why the productivity gap between AI power users and everyone else is widening

4. Why “models will eat your scaffolding for breakfast”

5. Why the next 12 to 24 months are a rare window where engineers can leap ahead before the role fully transforms

—

Brought to you by:

DX—The developer intelligence platform designed by leading researchers

Sentry—Code breaks, fix it faster

Datadog—Now home to Eppo, the leading experimentation and feature flagging platform

—

Episode transcript: https://www.lennysnewsletter.com/p/engineers-are-becoming-sorcerers

—

Archive of all Lenny's Podcast transcripts: https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&st=ahz0fj11&dl=0

—

Where to find Sherwin Wu:

• X: https://x.com/sherwinwu

• LinkedIn: https://www.linkedin.com/in/sherwinwu1

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• X: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

—

In this episode, we cover:

(00:00) Introduction to Sherwin Wu

(03:10) AI’s role in coding at OpenAI

(06:53) The future of software engineering with AI

(12:26) The stress of managing agents

(15:07) Codex and code review automation

(19:29) The changing role of engineering managers

(24:14) The one-person billion-dollar startup

(31:40) Management lessons

(37:28) Challenges and best practices in AI deployment

(43:56) Hot takes on AI and customer feedback

(48:57) Building for future AI capabilities

(50:16) Where models are headed in the next 18 months

(53:35) Business process automation

(57:22) OpenAI’s ecosystem and platform strategy

(01:00:50) OpenAI’s mission and global impact

(01:05:21) Building on OpenAI’s API and tools

(01:08:16) Lightning round and final thoughts

—

Referenced:

• Codex: https://openai.com/codex

• OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil (CPO at OpenAI, ex-Instagram, Twitter): https://www.lennysnewsletter.com/p/kevin-weil-open-ai

• OpenClaw: https://openclaw.ai

• The creator of Clawd: “I ship code I don’t read”: https://newsletter.pragmaticengineer.com/p/the-creator-of-clawd-i-ship-code

• The Sorcerer’s Apprentice: https://en.wikipedia.org/wiki/The_Sorcerer%27s_Apprentice_(Dukas)

• Quora: https://www.quora.com

• Marc Andreessen: The real AI boom hasn’t even started yet: https://www.lennysnewsletter.com/p/marc-andreessen-the-real-ai-boom

• Sarah Friar on LinkedIn: https://www.linkedin.com/in/sarah-friar

• Sam Altman on X: https://x.com/sama

• Nicolas Bustamante’s “LLMs Eat Scaffolding for Breakfast” post on X: https://x.com/nicbstme/status/2015795605524901957

• The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

• Overton window: https://en.wikipedia.org/wiki/Overton_window

• Developers can now submit apps to ChatGPT: https://openai.com/index/developers-can-now-submit-apps-to-chatgpt

• Responses: https://platform.openai.com/docs/api-reference/responses

• Agents SDK: https://platform.openai.com/docs/guides/agents-sdk

• AgentKit: https://openai.com/index/introducing-agentkit

• Ubiquiti: https://ui.com

• Jujutsu Kaisen on Crunchyroll: https://www.crunchyroll.com/series/GRDV0019R/jujutsu-kaisen?srsltid=AfmBOoqvfzKQ6SZOgzyJwNQ43eceaJTQA2nUxTQfjA1Ko4OxlpUoBNRB

• eero: https://eero.com

• Opendoor: https://www.opendoor.com

—

Recommended books:

• Structure and Interpretation of Computer Programs: https://www.amazon.com/Structure-Interpretation-Computer-Programs-Engineering/dp/0262510871

• The Mythical Man-Month: Essays on Software Engineering: https://www.amazon.com/Mythical-Man-Month-Software-Engineering-Anniversary/dp/0201835959

• There Is No Antimemetics Division: A Novel: https://www.amazon.com/There-No-Antimemetics-Division-Novel/dp/0593983750

• Breakneck: China’s Quest to Engineer the Future: https://www.amazon.com/Breakneck-Chinas-Quest-Engineer-Future/dp/1324106034

• Apple in China: The Capture of the World’s Greatest Company: https://www.amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

—

Lenny may be an investor in the companies discussed.