When stacked A/B wins don’t add up

On the podcast: about the cost of not tracking your experiments and decisions, how refunds and chargebacks quietly erase your paywall wins, and why stacking A/B test wins should compound your growth, but almost never does.

This conversation is shorter than usual and will be featured in RevenueCat’s State of Subscription Apps report. Each episode in this series will explore one crucial topic and share actionable insights from top subscription app operators.

Top Takeaways:

💸 Map your revenue history before running new experiments
Chart revenue across new subscribers, upgrades, renewals, and win-backs over time. Matching spikes and dips to past decisions reveals what actually moved the business and prevents you from re-learning expensive lessons.

🤫 Refunds and chargebacks are silent killers
A paywall “win” can quickly become a net negative if you aren’t tracking the downstream effects of cancellations, refunds, and chargebacks, which often hide the true cost of a seemingly successful experiment.

📈If your A/B test wins aren't showing up in top-line growth, something is wrong
Stacking 5% and 10% experiment wins should compound, but many teams see modest growth despite a long list of "winners". Set calendar reminders to recheck winning cohorts at 3 and 6 months, especially for price changes, to catch lifts that don't hold.

About Sara Grana: 

 🚀 Revenue Strategy Lead at Yousician, a revolutionary music platform for anyone to learn, play, create, and teach music. 

👋 LinkedIn

Follow us on X: 

• David Barnard - @drbarnard
• Jacob Eiting - @jeiting
• RevenueCat - @RevenueCat
• SubClub - @SubClubHQ

Episode Highlights:

[0:00] Introduction to Sara Grana, Revenue Strategy Lead at Yousician
[1:05] The importance of tracking experiments and business decisions in subscription apps
[2:19] Mapping revenue and understanding its evolution across different user segments
[3:06] Tracking revenue changes and connecting them to business decisions
[4:34] The pitfalls of focusing too much on early funnel metrics and ignoring long-term impacts
[5:26] The impact of chargebacks and refunds on paywall performance and customer retention
[7:20] Why understanding downstream effects is crucial for making smart pricing decisions
[8:44] The challenges and opportunities of introducing new subscription plans (e.g., lifetime subscriptions)
[9:43] How commercial strategy influences churn rates and renewals
[13:13] The importance of rechecking experiments after months to measure long-term impact
[14:52] Sara's advice on when to revisit experiments based on their impact on pricing and user behavior
[15:49] Tracking cohort data for subscription retention and understanding renewal trends
[16:21] Why surprising lifts in experiments may require deeper investigation
[17:13] The mismatch between short-term experiment results and long-term growth expectations
[18:02] Final thoughts on driving sustainable growth, tracking, and adapting strategies over time