Audit step for surveys: random checks

Today’s Episode

Discovery might be the most important core PM skill for building great products.

But most PMs are unprepared to do discovery in AI. PMs run surveys incorrectly, conduct interviews poorly, and end up with poor insights.

Today will give you the roadmap to avoid all those mistakes.

Caitlin Sullivan is a user research expert who runs courses teaching PMs how to do AI-powered discovery. And in today’s episode, she shows you exactly how she does it.

We’re talking live demos. Step-by-step workflows. Real survey data. Real interview transcripts.

This is a masterclass in discovery. The kind that moves the needle.

----

Brought to you by:

Maven: Get 15% off Caitlin’s courses with code AAKASHxMAVEN

Pendo: The #1 software experience management platform

Jira Product Discovery: Plan with purpose, ship with confidence

Kameleoon: AI experimentation platform

Amplitude: The market-leader in product analytics

----

Key Takeaways:

1. Replicate the human process - Good AI analysis mirrors how experienced researchers work: comb through data first, then synthesize. Never jump straight to "give me themes."2. Use multi-step prompting - Load context in one prompt, run per-participant analysis in the next, then verify. Cramming everything into one prompt degrades quality.3. Code before you count - For surveys, apply inductive coding labels to every response before asking for patterns. Skipping this step leads to miscategorized, unreliable results.4. Always audit AI's work - Force the model to re-check its own analysis. It catches contradictions, overexaggerated intensity ratings, and miscoded responses regularly.5. Claude wins on nuance, Gemini wins on frequency - Claude gives more thorough, complete analysis by default. Gemini surfaces top-frequency themes faster but misses smaller patterns.6. Define everything explicitly - Quotes, ratings, emotional intensity levels, contradiction types. If you assume the model shares your definitions, you'll get inconsistent results.7. Markdown files beat raw transcripts - Converting transcripts to structured markdown improves accuracy and helps you work around token limits on non-Max plans.8. Parallelize with Claude Code agents - Set up agent markdown files for interview and survey analysis, then run both simultaneously. Cuts total analysis time in half again.

----

Related Content

Newsletters:

How to Do Product Discovery Right

Advanced Techniques: Continuous Discovery

Customer Interviews: Advanced Techniques

Podcasts:

Teresa Torres’ Guide to AI Discovery

Complete Course: AI Product Discovery

Ultimate Guide to Knowing Your Users as a PM

----

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

If you want to advertise, email productgrowthppp at gmail.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.news.aakashg.com/subscribe