Lenny's Podcast: Product | Career | Growth

chevron_right

Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon

whatshot 1983 snips

Jan 11, 2026

Guest

Kiriti Badam

Guest

Aishwarya Naresh Reganti

Aishwarya Naresh Reganti, an AI researcher and product builder, and Kiriti Badam, an engineer with experience at OpenAI, share insights from deploying over 50 AI products. They discuss the crucial differences between AI and traditional software, emphasizing a gradual increase in autonomy for successful products. Reliability emerges as a key blocker in enterprise adoption, while customer trust and effective monitoring methodologies are highlighted as essential. They also stress the importance of design judgment and persistent problem-solving in the evolving landscape of AI.

01:26:22

forum

Ask episode

web_stories

AI Snips

view_agenda

Chapters

menu_book

Books

auto_awesome

Transcript

info_circle

Episode notes

Reliability Is The Enterprise Bottleneck

Reliability is the top enterprise blocker to deploying AI; many enterprises avoid customer-facing AI products due to trust concerns.
This explains why current enterprise AI focuses on productivity (low-autonomy) tools rather than fully autonomous agents.

The Three Pillars Of AI Success

Successful AI efforts combine three pillars: committed leaders, empowering culture, and technical obsession with workflows and data quality.
Leaders must relearn intuitions, be hands-on, and foster collaboration between PMs, engineers, and subject-matter experts.

Use Evals Plus Production Monitoring

Combine evals (predefined evaluation datasets/metrics) with production monitoring; neither alone suffices to catch all failures.
Use implicit signals (regenerations, opt-outs, usage drops) to surface traces needing deeper evaluation datasets.

Get the Snipd Podcast app to discover more snips from this episode

What builders see in 2025 AI adoption

Two key AI differences from traditional software

05:08 • 7min

chevron_right

Start small: increment autonomy over time

11:39 • 2min

chevron_right

Customer support example: progression from routing to autonomy

13:39 • 2min

chevron_right

Behavior calibration and constraining autonomy

16:02 • 4min

chevron_right

Why low-autonomy iterations force problem-first thinking

19:45 • 2min

chevron_right

Reliability as the enterprise blocker

21:31 • 2min

chevron_right

Prompt injection and security risks

23:32 • 2min

chevron_right

Success triangle: leaders, culture, technical obsession

25:43 • 8min

chevron_right

The evals vs. monitoring debate

33:21 • 5min

chevron_right

What 'evals' actually mean in practice

38:13 • 3min

chevron_right

How Codex balances evals and customer feedback

Continuous Calibration / Continuous Development (CC/CD)

46:00 • 5min

chevron_right

CC/CD applied: customer support stages and flywheel

51:19 • 6min

chevron_right

When to advance stages and staying vigilant

57:17 • 4min

chevron_right

Overhyped and under-hyped AI trends

01:01:24 • 3min

chevron_right

Design, taste, and product judgment matter most

01:04:13 • 1min

chevron_right

Product vision for 2026: proactive and multimodal agents

01:05:26 • 3min

chevron_right

Skills to focus on: judgment and persistence

01:08:53 • 6min

chevron_right

Final advice: obsess over customers and workflows

01:14:25 • 53sec

chevron_right

Lightning round: books, shows, products, mottos

01:15:19 • 9min

chevron_right

Where to follow their work and course details

• Mentioned in 37 episodes

When Breath Becomes Air

E. Z. - Reader

In this profoundly moving memoir, Paul Kalanithi chronicles his transformation from a medical student to a neurosurgeon at Stanford, and finally to a patient confronting his own mortality. Diagnosed with stage IV lung cancer at 36, Kalanithi grapples with fundamental philosophical questions about what makes a life worth living in the face of death. The book is a reflection on the challenge of facing death and the relationship between doctor and patient, written with eloquence and intelligence. Kalanithi's writing delves into his lifelong love of literature and his attempt to connect the sciences and humanities to gain a greater understanding of humanity.

#2116

• Mentioned in 21 episodes

A Fire upon the Deep

Vernor Vinge

In 'A Fire Upon the Deep', Vernor Vinge explores a vast universe divided into 'zones of thought' where the potential of intelligent life varies greatly. The story begins with a human expedition that awakens a destructive artificial intelligence, the Blight, which threatens to destroy entire civilizations. Two children, Jefri and Johanna Olsndot, escape the initial destruction but are captured by the Tines, an intelligent dog-like alien species with a hive mind. Meanwhile, Ravna Bergsndot, an intergalactic librarian, and her companions embark on a perilous rescue mission to save the children and recover an artifact that could stop the Blight. The novel is renowned for its complex world-building, unique alien cultures, and blend of hard science fiction and high drama.

#133

• Mentioned in 146 episodes

The Three-Body Problem

Cixin Liu

Cixin Liu's 'The Three-Body Problem' is a critically acclaimed science fiction novel that blends hard science with compelling storytelling. The story follows a group of scientists who make contact with an alien civilization, leading to unforeseen consequences for humanity. Liu's intricate world-building and exploration of complex scientific and philosophical themes have earned him widespread recognition. The novel is known for its ambitious scope and its exploration of the potential challenges and dangers of encountering extraterrestrial life. It's a thought-provoking work that has captivated readers worldwide.

Aishwarya Naresh Reganti and Kiriti Badam have helped build and launch more than 50 enterprise AI products across companies like OpenAI, Google, Amazon, and Databricks. Based on these experiences, they’ve developed a small set of best practices for building and scaling successful AI products. The goal of this conversation is to save you and your team a lot of pain and suffering.

We discuss:

1. Two key ways AI products differ from traditional software, and why that fundamentally changes how they should be built

2. Common patterns and anti-patterns in companies that build strong AI products versus those that struggle

3. A framework they developed from real-world experience to iteratively build AI products that create a flywheel of improvement

4. Why obsessing about customer trust and reliability is an underrated driver of successful AI products

5. Why evals aren’t a cure-all, and the most common misconceptions people have about them

6. The skills that matter most for builders in the AI era

—

Brought to you by:

Merge—The fastest way to ship 220+ integrations: https://merge.dev/lenny

Strella—The AI-powered customer research platform: https://strella.io/lenny

Brex—The banking solution for startups: https://www.brex.com/product/business-account?ref_code=bmk_dp_brand1H25_ln_new_fs

—

Transcript: https://www.lennysnewsletter.com/p/what-openai-and-google-engineers-learned

—

My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/183007822/referenced

—

Get 15% off Aishwarya and Kiriti’s Maven course, Building Agentic AI Applications with a Problem-First Approach, using this link: https://bit.ly/3V5XJFp

—

Where to find Aishwarya Naresh Reganti:

• LinkedIn: https://www.linkedin.com/in/areganti

• GitHub: https://github.com/aishwaryanr/awesome-generative-ai-guide

• X: https://x.com/aish_reganti

—

Where to find Kiriti Badam:

• LinkedIn: https://www.linkedin.com/in/sai-kiriti-badam

• X: https://x.com/kiritibadam

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• X: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

—

In this episode, we cover:

(00:00) Introduction to Aishwarya and Kiriti

(05:03) Challenges in AI product development

(07:36) Key differences between AI and traditional software

(13:19) Building AI products: start small and scale

(15:23) The importance of human control in AI systems

(22:38) Avoiding prompt injection and jailbreaking

(25:18) Patterns for successful AI product development

(33:20) The debate on evals and production monitoring

(41:27) Codex team’s approach to evals and customer feedback

(45:41) Continuous calibration, continuous development (CC/CD) framework

(58:07) Emerging patterns and calibration

(01:01:24) Overhyped and under-hyped AI concepts

(01:05:17) The future of AI

(01:08:41) Skills and best practices for building AI products

(01:14:04) Lightning round and final thoughts

—

Referenced:

• LevelUp Labs: https://levelup-labs.ai/

• Why your AI product needs a different development lifecycle: https://www.lennysnewsletter.com/p/why-your-ai-product-needs-a-different

• Booking.com: https://www.booking.com

• Research paper on agents in production (by Matei Zaharia’s lab): https://arxiv.org/pdf/2512.04123

• Matei Zaharia’s research on Google Scholar: https://scholar.google.com/citations?user=I1EvjZsAAAAJ&hl=en

• The coming AI security crisis (and what to do about it) | Sander Schulhoff: https://www.lennysnewsletter.com/p/the-coming-ai-security-crisis

• Gajen Kandiah on LinkedIn: https://www.linkedin.com/in/gajenkandiah

• Rackspace: https://www.rackspace.com

• The AI-native startup: 5 products, 7-figure revenue, 100% AI-written code | Dan Shipper (co-founder/CEO of Every): https://www.lennysnewsletter.com/p/inside-every-dan-shipper

• Semantic Diffusion: https://martinfowler.com/bliki/SemanticDiffusion.html

• LMArena: https://lmarena.ai

• Artificial Analysis: https://artificialanalysis.ai/leaderboards/providers

• Why humans are AI’s biggest bottleneck (and what’s coming in 2026) | Alexander Embiricos (OpenAI Codex Product Lead): https://www.lennysnewsletter.com/p/why-humans-are-ais-biggest-bottleneck

• Airline held liable for its chatbot giving passenger bad advice—what this means for travellers: https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know

• Demis Hassabis on LinkedIn: https://www.linkedin.com/in/demishassabis

• We replaced our sales team with 20 AI agents—here’s what happened | Jason Lemkin (SaaStr): https://www.lennysnewsletter.com/p/we-replaced-our-sales-team-with-20-ai-agents

• Socrates’s quote: https://en.wikipedia.org/wiki/The_unexamined_life_is_not_worth_living

• Noah Smith’s newsletter: https://www.noahpinion.blog

• Silicon Valley on HBO Max: https://www.hbomax.com/shows/silicon-valley/b4583939-e39f-4b5c-822d-5b6cc186172d

• Clair Obscur: Expedition 33: https://store.steampowered.com/app/1903340/Clair_Obscur_Expedition_33/

• Wisprflow: https://wisprflow.ai

• Raycast: https://www.raycast.com

• Steve Jobs’s quote: https://www.goodreads.com/quotes/463176-you-can-t-connect-the-dots-looking-forward-you-can-only

—

Recommended books: