Daliana's Game Stop abusing A/B testing, toxic experimentation culture, how to run A/B tests with rigor - Che Sharma - The Data Scientist Show #071
17 snips
Nov 4, 2023 Che Sharma, former data scientist at Airbnb and founder of Eppo, discusses toxic behaviors in experimentation culture, A/B testing best practices, and A/B testing for ML models on The Data Scientist Show. Topics include statistical power, effect size, monitoring metrics, alternative methods to A/B testing, difference in differences method, and A/B testing in ML and AI.
AI Snips
Chapters
Transcript
Episode notes
Guard Rails And Prolong Tests If Needed
- Add automated guardrails that alert when key metrics drop during experiments.
- If guardrails look concerning, continue running the test longer before deciding.
Make Reports Focused And Consistent
- Report a narrow set of core metrics decided by leadership and put explorations in an appendix.
- Avoid cherry-picking slices as headline wins; keep exploratory cuts for idea generation.
Design Reports Leaders Will Read
- Publish experiment reports where leaders actually live (Slack, email, Notion) and keep format consistent.
- Include context, screenshots, and a one‑page core-metrics headline before deeper analysis.