Complete Course: AI Experimentation

34 snips

Aug 29, 2025

Frederic De Todaro, the Chief Product Officer at Kameleoon, shares insights from over 12 years of driving AI experimentation. He discusses how AI has revolutionized the experimentation landscape, highlighting its role in speeding up the build process and fostering collaboration among teams. Frederic breaks down the evolution of AI tools, contrasting traditional A/B testing with advanced methods like multi-arm bandits. He also dives into vibe coding's impact on rapid prototyping, and offers tips on measuring AI feature success, emphasizing the importance of clear metrics.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ADVICE

Target Users With AI Scores

Use AI targeting scores instead of manually building segments to decide who sees which experience.
Let the model predict conversion intent and target offers to users with higher likelihood to convert.

INSIGHT

Bandits: Speed vs Personalization

Multi-armed bandits optimize for performance by shifting traffic to better variants quickly.
Contextual bandits personalize per user by learning which variant works best for each individual.

ADVICE

Salvage Failed Tests With Opportunity Detection

Use AI opportunity detection to automatically drill down into non-conclusive experiments.
Let AI surface segments (e.g., mobile users) where a variation actually performs well so you can salvage wins.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

"AI has been the biggest driver of change in experimentation I've seen in my career."

That's Frederic De Todaro, Chief Product Officer at Kameleoon (profitable SaaS with 2K+ customers).

Fred has been at Kameleoon for 12+ years. In that role, he's helped thousands of teams use AI to experiment faster and smarter.

In today’s episode he’s breaking down:

How AI changes experimentation

How to experiment with AI features

Last week, I covered how one aspect of this: vibe experimentation. Today’s video is the A to Z AI impact.

If you experiment at work, this episode is for you.

----

Check out the conversation on Apple, Spotify and YouTube.

* Mobbin: Discover real-world design inspiration

* Jira Product Discovery: Build the right thing, reliably

* AI Product Strategy Certificate for Leaders: Get $550 off

----

Timestamps:

00:00 How AI Changed Experimentation Overview

01:54 The 4 Steps of Experimentation Framework

14:12 ADS

16:00 How AI has Changed Experimentation

21:08 User Behaviour Models

26:56 Multi-Armed Bandit vs Contextual Bandit

30:05 ADS

31:55 AI Content Genration

35:13 How Vibe Coding Changes Experimentation

41:35 Live Demo From Idea to Running Experiment in 2 Minutes

43:36 Two-Minute Build Achievement

51:49 How to Measure AI Features Properly

54:17 Measuring RAG Systems 3 Key Metrics

01:07:18 Best Experimentation Company Booking.com

01:10:10 Biggest PM Mistakes in Experimentation

01:13:52 Ending

----

Key Takeaways

1. Build is the bottleneck. Most teams can't A/B test because developers are busy. AI removes this constraint anyone can now create experiments in minutes using plain English.

2. 80% of experiments fail. But with AI opportunity detection, you can drill into failed experiments to find hidden wins, like features that work great on mobile but fail on desktop.

3. Vibe coding meets experimentation. It's not enough to build prototypes quickly. You need to test them with real users at scale. Prompt-based experimentation bridges this gap.

4. Context is everything. AI performs best when it has access to your website's framework, design system, and past experiments. The more context, the better the ideas and implementations.

5. Humans still matter. PMs bring business context, data scientists ensure statistical rigor, and AI handles the grunt work. It's augmentation, not replacement.

6. Start simple with feature flags. You don't need to copy Booking.com overnight. Begin with feature flags, then rollouts, then full experimentation. AI makes each step easier.

7. Measure beyond usage. For AI features, track: How many prompts to success? Time from idea to live? How often do developers step in? These reveal true value.

8. Multi-armed bandits for speed, contextual for personalization. Use multi-armed when you need quick answers. Use contextual when personalizing for each user.

9. Discovery and experimentation are partners. Discovery tells you what users say they want. Experimentation tells you what they actually do. You need both for the full picture.

----

Check out the conversation on Apple, Spotify and YouTube.

----

Related Podcasts:

* How to Build Things Faster as a Product Team

* Lessons from Super-Senior IC Experimentation PM

* Amplitude CEO: Demo, Story, and How They Build Product

-----

P.S. More than 85% of you aren't subscribed yet. If you can subscribe on YouTube, follow on Apple & Spotify, my commitment to you is that we'll continue making this content better.

----

If you want to advertise, email productgrowthppp at gmail.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.news.aakashg.com/subscribe