Experiment Nation: The Podcast S3E21 - A/B Testing Statistics Concepts Experimenters must know with Ronny Kohavi
53 snips
Jun 4, 2023 Data scientist Ronny Kohavi discusses A/B testing statistics concepts all experimenters must know, including the Overall Evaluation Criterion, Twyman's law, and the amount of traffic needed for A/B tests. He shares insights on handling tests that turn out to be wrong and presenting experiment results to the CEO of Amazon. The podcast also highlights the benefits of AV tests and variety in designs for experimentation and CRO, while cautioning against the pitfalls of long-term testing.
AI Snips
Chapters
Transcript
Episode notes
Most Experiment Ideas Fail More Often Than You Think
- Most A/B test ideas fail frequently, revealing that teams overestimate idea quality.
- Ronny Kohavi observed 60–70% failure at Microsoft, ~85% at Bing, and 92% at Airbnb, showing failure is widespread even in big tech.
Automate Trust Tests In Your Experiment Platform
- Build automated trust checks into your experimentation platform to detect common bugs and validate results.
- Kohavi recommends SRM checks, A/A tests, p-value uniformity tests, and correct variance methods for ratio metrics.
Removing Jeff Bezos's Favorite Feature Turned Out Better
- Jeff Bezos loved bottom-of-page deals, but removing them improved performance and won the experiment.
- Kohavi recounts removing the slow feature and Bezos accepted the data-driven recommendation to remove it.