The Self-Improving Flaky Spec Skill

Brian Scanlan is a senior principal engineer at Intercom, where he’s led the company’s transformation to AI-first engineering. In just nine months, Intercom doubled their R&D throughput while maintaining code quality, with 100% of engineers—plus designers, PMs, and TPMs—now shipping code via Claude Code.

What you’ll learn:

1. How Intercom doubled their merged PRs per R&D employee in just nine months using Claude Code
2. The telemetry infrastructure they built to measure AI adoption and quality across hundreds of engineers
3. Why they built a skills repository with hooks that enforce engineering standards automatically
4. How they’re preparing their product for an agent-first world with CLIs, MCPs, and ephemeral APIs
5. The permission and accountability framework that enabled rapid AI adoption
6. Why backlog zero is now achievable and what that means for engineering culture

—

Brought to you by:

Celigo—Intelligent automation built for AI

Cursor—The best way to code with AI

—

In this episode, we cover:

(00:00) Introduction to Brian Scanlan

(02:40) Why Intercom went all-in on AI for both product and engineering

(05:01) The breakthrough moment with Opus 4.6 and Christmas break 2025

(07:02) Demo: Intercom’s merged PRs per R&D head

(12:50) Agent-first work as a fundamental reimagining of technical workflows

(14:27) The cost tradeoff: treating AI spend as an investment

(16:47) Measuring quality

(21:22) Demo: Shipping a redirect in the Rails monolith with Claude Code

(24:03) Creating a custom PR skill

(26:33) Building a software factory with predictable quality standards

(30:15) Telemetry infrastructure: Honeycomb for skill usage tracking

(32:10) Session data collection and personalized usage insights

(36:08) Quick overview

(39:20) Walking through Intercom’s skills repository

(42:16) Deep dive: The flaky spec skill and how it reached 100x capability

(46:44) The “and then” workflow for building comprehensive skills

(52:31) The live website and overview of workflows

(53:32) How internal AI experience informs customer product decisions

(56:18) Making SaaS products agent-friendly with CLIs and helpful hints

(01:03:49) Why conversion drop-off is invisible in agent-driven workflows

(01:05:28) Lightning round and final thoughts

—

Detailed workflow walkthroughs from this episode:

• How Intercom Doubled Engineering Output: Brian Scanlan's 4 AI Workflows for Claude Code: https://www.chatprd.ai/how-i-ai/how-intercom-doubled-engineering-output-brian-scanlan-ai-workflows-for-claude-code

• Design an Agent-Friendly CLI to Automate SaaS Product Onboarding: https://www.chatprd.ai/how-i-ai/workflows/design-an-agent-friendly-cli-to-automate-saas-product-onboarding

• Build a Self-Improving AI Agent to Automatically Fix Flaky Tests: https://www.chatprd.ai/how-i-ai/workflows/build-a-self-improving-ai-agent-to-automatically-fix-flaky-tests

• Automate High-Quality Pull Request Descriptions with a Custom AI Skill: https://www.chatprd.ai/how-i-ai/workflows/automate-high-quality-pull-request-descriptions-with-a-custom-ai-skill

—

Tools referenced:

• Claude Code: https://claude.ai/code

• Cursor: https://cursor.com/

• Honeycomb: https://www.honeycomb.io/

• Snowflake: https://www.snowflake.com/

• Fin AI: https://www.intercom.com/fin

• Vercel: https://vercel.com/

—

Other references:

• Intercom GitHub Repo: https://github.com/intercom

• Google API Go Client Repo: https://github.com/googleapis/google-api-go-client

—

Where to find Brian Scanlan:

X: https://x.com/brian_scanlan

LinkedIn: https://www.linkedin.com/in/scanlanb/

Company: https://www.intercom.com

—

Where to find Claire Vo:

ChatPRD: https://www.chatprd.ai/

Website: https://clairevo.com/

LinkedIn: https://www.linkedin.com/in/clairevo/

X: https://x.com/clairevo

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email jordan@penname.co.