To help us with this topic, Alex Mitchell is our guest. He is Director of Product at Kin Insurance and founder of The Modern Product Manager, which provides resources and courses for aspiring product managers. He also is involved in other organizations, lending his product expertise.

Summary of some concepts discussed for product managers

[3:47] Tell us about experimentation and testing.

We need to have a broad definition of experimentation; it’s more than controlled experiments like A/B testing. Experimentation is a foundational and continuous element of product management. Don’t wait until you’ve built a feature to experiment. Test continuously throughout your product development lifecycle. Do small experiments to learn more and continually develop hypotheses you can test to help you make better decisions.

[6:19] Product managers often become experts in their industry, and that can lead to assumptions. How do you test assumptions?

Assumptions slow down innovation. One way to counter that is to involve more people in experimentation. Including your team and people outside your team will help keep innovation going and prevent you from getting complacent. Second, move around to different domains in your company if you can. This will help keep your ideas fresh and keep you coming up with testing and experimentation ideas.

[8:25] What’s an example of using experimentation at Kin Insurance?

At Kin Insurance, we sell home insurance to catastrophe regions like Florida and Louisiana. Through research, we observed that many customers were not making changes to their insurance after their initial purchase, like adding additional coverage, even if changes were in their best interest. We hypothesized they didn’t know they could make changes and if we let them know about additional products soon after their initial purchase, a meaningful percentage of people would at least be interested in learning more.

If we were not taking an experimental approach to this, we might design a full-blown, intricate add-on system. We wanted the opposite since we were trying to validate whether our customers were interested. We took a simple approach and found a subset of customers eligible for flood insurance. A week after their initial purchase, we sent them an email conveying the benefits of flood insurance and asking if they were interested, and we found that many people were. We tested our hypothesis in a very short period, with a very small amount of effort. With testing like this, you can validate or invalidate your hypothesis, and you’ll learn other things along the way too. We did all this with no development effort and didn’t invest any of our most precious resources until we had validated our hypothesis and understood the customer experience well.

You can use lightweight, scrappy tests to get information quickly and inexpensively. However, this may be different from how some of your colleagues are used to working. They might want a fully-developed plan, while you’re trying to do something you’ve never done before. Be prepared for skepticism, questions, and discomfort—that may be an indication you’re on the right track.

[17:46] Have you seen an experiment produce unhelpful data? Why was that?

One reason an experiment can produce unhelpful data is if it doesn’t line up with the hypothesis you’re trying to test or if you don’t have a way to make changes to prove or disprove your hypothesis. Another reason is if you’re looking at data that’s irrelevant to your experiment or attributing changes to your experiment that were actually caused by something else. Make sure you have someone on your team who understands statistical significance and what is likely to be noise or the result of an unrelated variable.

[23:07] What key tip about experimenting would you like to leave us with?

Your tests should be small, focused, and easy to explain. If you can’t explain in 15 seconds what you’re trying to learn and how you’re going to accomplish it, your test is probably too big. Staying focused gives you a better opportunity to learn and get others involved.

Action Guide: Put the information Alex shared into action now. Click here to download the Action Guide.

Useful links:

Read Alex’s articles on Medium
Visit Alex’s website, TheModernProductManager.com
Connect with Alex on LinkedIn

Innovation Quote

“Failure is an option here. If things are not failing, you are not innovating enough.” – Elon Musk

Thanks!

Thank you for taking the journey to product mastery and learning with me from the successes and failures of product innovators, managers, and developers. If you enjoyed the discussion, help out a fellow product manager by sharing it using the social media buttons you see below.

Source