Daliana's Game A/B testing at Airbnb, building next-gen experimentation platform at Eppo - Che Sharma - The Data Scientist Show #068
31 snips
Aug 25, 2023 Che Sharma, former data scientist at Airbnb and founder of Eppo, talks about A/B testing best practices, A/B testing for ML models, and his career journey. They discuss successful A/B testing, interpreting and communicating test results, A/B testing best practices for ML models, centralizing experiment analysis, preparing data scientists for the future, developing communication skills, transitioning to a manager role, and the future of experimentation.
AI Snips
Chapters
Transcript
Episode notes
New Tab Boosted Airbnb Bookings
- Opening listings in new tabs at Airbnb, despite design opposition, led to the largest uplift in bookings.
- This simple UI change preserved search context and boosted user engagement effectively.
Master Experiment Statistical Basics
- Run power analyses to ensure experiments have enough data to detect effects.
- Use sample ratio mismatch tests and robust randomization to avoid biases and errors.
Start Small to Build Trust
- Build strong human relationships to align teams on experimentation best practices.
- Start with small experiments to build muscle and infrastructure confidence before big projects.