Ecommerce Playbook: Numbers, Struggles & Growth Turning Incrementality Tests Into Action That Makes You Money
Feb 26, 2026
George Davis, CMO at Cozy Earth, shares hands-on experience running holdout tests for a DTC home goods brand. Olivia Kory, Chief Strategy Officer at Haus, designs and operationalizes incrementality experiments. They discuss when to run holdouts, why results shift month-to-month, how to turn iROAS into actionable IMR, practical levers to improve tests, and how to use incrementality factors for real budget decisions.
AI Snips
Chapters
Transcript
Episode notes
Why Click Attribution Breaks As Brands Mature
- As brands scale, click-based signals degrade because distribution, repeat customers, and non-click channels grow.
- Taylor maps the typical progression: early trust in platform ROAS, then MTA, then holdout testing as channels and organic mix expand.
Always Use Holdouts To Prove Causation
- Do use a randomized holdout to measure true incrementality because you need a counterfactual.
- Olivia: turn off the channel for a test cell or region rather than only doing spend-up/spend-down experiments to establish causation.
Make Marketing Metrics Follow Your Financial Goal
- Align marketing KPIs to the company's financial objective and measure incremental positive contribution margin.
- Taylor: decide whether shareholders want free cash flow or P&L profit, then set IMR/thresholds for media accordingly.