Apptivate: App Marketing Explained Data Science: Designing an Experimentation Platform - Shan Huang (Zalando)
Today’s guest is Shan Huang, the Senior Applied Scientist at Zalando, a multinational e-commerce platform for shoes and fashion. Shan is also the co-founder of the German-Chinese Association for Artificial Intelligence, a nonprofit advancing the exchange of education, research, and public resources between Germany and China in the field of AI.
Questions Shan Answered in this Episode:
- Can you give me an explanation of what an experimentation platform is? What is an example of how it’s used?
- How do you set the limitations? How do you define what can be experimented?
- What are the biggest challenges to building such a platform?
- If you could go back in time and could give yourself one hint or remove one obstacle in building this platform, what would that be?
- Are you running automated optimization a/b tests?
- Are there any tricks to increase the efficiency or decrease the runtime of the experimentation?
- How do you support people knowing what experiments to run, what’s interesting, possible to test, etc?
- What was the reason for creating the experimentation platform?
Timestamp:
- 2:53 The many use cases of Zalando’s experimentation platform
- 6:45 Putting together the right team
- 10:19 What’s important in the beginning
- 11:27 Hypothesis testing methodology
- 13:14 Adaptive experimentation
- 15:03 Methods for improving experimentation efficiency
- 18:44 Setting up a process for running a/b tests
- 23:04 Power to the product team
Quotes:
(6:50-6:57) “I think one of the biggest challenges is that building this kind of platform requires a team of different experts in different domains.”
(10:31-10:56) “In the beginning it’s about providing infrastructure and also helping our stakeholders with other teams learn a/b testing, understand a/b testing, because statistics is sometimes a very confusing thing--confidence interval, significance--it’s not so easy to explain. And I think it might be helpful to get a solid groundwork on this stuff.”