Case studies of scrappy product management experiments

Watch on YouTube

TLDR

In this episode, I’m interviewing Jeff Lash, VP of Product Management at Insperity, to demystify product experimentation for product managers. Jeff unpacks scrappy ways to test assumptions, mitigate risk, and maximize learning, sharing case studies from his work in B2B product management. We discuss real examples, key principles for experimentation, and navigating organizational dynamics to drive informed product decisions.

Introduction

Most product managers think experimentation requires expensive A/B testing software, a team of data scientists, and thousands of users. They’re wrong. You can and should be testing your riskiest assumptions today, and doing so in ways that are fast and frugal. By the end of this episode, you’ll have a toolkit of testing methods that you can deploy immediately.

Our guest is the perfect guide for this. Jeff Lash is the Vice President of Product Management at Insperity. Before that, he spent nearly a decade at Forrester and SiriusDecisions, where he advised the world’s top product organizations on exactly these strategies. He is the author of the long-running How To Be A Good Product Manager blog and a product management veteran who has transitioned from practitioner to researcher, analyst, and adviser, and then back to the front lines of product leadership.

Summary of Concepts Discussed for Product Managers

The Purpose of Experimentation:
Experimentation prevents product managers from jumping to solutions by validating that they’re solving the right problems with the right solutions. Jeff emphasizes that effective experimentation requires humility and an openness to learning. This approach helps avoid costly mistakes of building products based on unverified assumptions and mitigates business risk.

Fast, Frugal Experiments:
Jeff explains that experiments should deliver maximum learning for minimum investment. Experiments should be built upon foundational customer research and always include measurable objectives. He reminds product managers not to rely solely on digital tools, especially in B2B contexts where the customer base is smaller and sales cycles are longer.

Case Studies of Product Experimentation:
Through several case studies, Jeff Lash illustrates experimentation methods:

• Using mock-ups for concept testing: Before building a new data-reporting SaaS, a product team manually created mock-up sample reports and pitched them to clients. The low demand they discovered helped avoid unnecessary development.
• Sales-Driven Product Testing: Collaborating with sales, an organization defined clear success metrics, launched a pilot with a limited customer group, and used real buying signals (not just sales enthusiasm) to validate new offerings, minimizing risk and maximizing buy-in.
• Content Access Limits: Unsure about the right threshold for content access in a subscription product, a company temporarily gave all customers unlimited access to gather data on which content they were accessing, later allowing them to set limits that balanced user delight and business goals.
• Testing with a Sales Presentation: In response to sales insisting there was a market for a new product, a product team created a sales pitch deck. After several meetings and pitches, they found zero customer interest, which revealed the real gap was not product, but access to the right buyer. This low-cost experiment saved significant time and resources by preventing the team from building an unwanted solution.

Navigating Organization Dynamics:
Not all experiments yield the result everyone wants. Jeff discusses how to align teams around experiment outcomes—even unpopular ones—and communicate evidence while managing executive or sales pressure. He stresses the importance of cross-functional alignment, especially in B2B, and framing experiments by the core questions they’re meant to answer.

B2B vs. B2C Experimentation:
While B2C may allow rapid, large-scale testing, B2B experimentation requires more coordination with sales, legal, and customer success to avoid customer confusion or contractual risks. Building internal buy-in and clear communication is critical for successful, reversible tests.

Useful Links

• Visit Jeff’s website
• Read Jeff’s blog, How To Be A Good Product Manager
• Connect with Jeff on LinkedIn
• Learn more about Insperity
• Listen to episode 127: B2B product management – with Jeff Lash

Innovation Quote

“It is not the most intellectual of the species that survives; it is not the strongest that survives; but the species that survives is the one that is able best to adapt and adjust to the changing environment in which it finds itself.” – Leon Megginson

Application Questions

1. What assumptions in your current product strategy could be tested with a simple experiment this quarter?
2. How does your team define success criteria for experiments? Who needs to be involved in that definition?
3. Have you ever faced resistance to experiment results? How did (or would) you handle those internal politics?
4. In what ways do you coordinate experiments with sales, marketing, and customer success to minimize customer and internal confusion?
5. How might you adapt these B2B-focused experimentation techniques for your context, whether B2B, B2C, or hybrid?

Bio

Jeff is VP, Product Management at Insperity, where he is responsible for enterprise product management and leading product strategy across this $6B+ public company that provides HR services to small and medium-sized businesses. He has spent over 20 years in product management, including as an advisor to product management leaders at Forrester/SiriusDecisions. For years he wrote the popular “How to Be a Good Product Manager” blog, and he is founder of the not-for-profit St. Louis Product Management Group and Chair of ProductCamp St. Louis. 

Thanks!

Thank you for taking the journey to product mastery and learning with me from the successes and failures of product innovators, managers, and developers. If you enjoyed the discussion, help out a fellow product manager by sharing it using the social media buttons you see below.

Source