Software Engineering Radio - the podcast for professional software developers SE-Radio Episode 360: Pete Koomen on A/B Testing
Mar 13, 2019
Pete Koomen, Co-founder and CTO at Optimizely, dives deep into A/B testing, sharing insights from running over 200,000 tests. He reveals how structured experimentation can elevate product innovation, highlighting the importance of hypotheses and success metrics. The conversation also addresses the challenges of testing, the significance of understanding user behavior, and the ethical considerations in experimentation. Koomen emphasizes fostering a culture of testing within organizations to drive data-driven decision-making and enhance user experience.
AI Snips
Chapters
Transcript
Episode notes
End Experiments Based On Data
- Use advanced statistical methods to analyze experiments in real time and decide early when to stop. - Avoid misusing t-tests by stopping experiments early; follow modern stats for digital contexts.
Organizational Challenges in Experimentation
- Experimentation requires overcoming organizational challenges beyond just technical hurdles. - Build culture, teams, and processes for running, analyzing, and prioritizing experiments effectively.
Select Your Test Audience Carefully
- Carefully choose who sees test variations to avoid user frustration and legal issues. - Use segmentation strategies like geographic isolation to minimize cross-group impact in networked systems.