The Andrew Faris Podcast He's Seen 1000s Of Split Tests. Here's What Almost Everyone Gets Wrong. (With Drew Marconi)
Mar 10, 2026
Drew Marconi, founder and CEO of IntelliGems and A/B testing expert, breaks down where most experiments go wrong. He explains why premature wins mislead, why longer, replicable tests matter, and how pricing, offers, and shipping thresholds move the needle. Practical testing strategy and automation get plenty of attention.
AI Snips
Chapters
Transcript
Episode notes
Temptation To Share Early Test Swings
- Andrew described impulse to screenshot early big swings and share premature wins on Slack or Twitter.
- He joked IntelliGems should lock screens or include an AI therapist to stop people from p-hacking early.
Replicability Problem In Split Testing
- Many split-test wins are not replicated and may be false positives, mirroring the academic replication crisis.
- Andrew argues that businesses often accept one-off results as truth without rerunning tests to confirm real effects over time.
Run Tests Long Enough To Tighten Estimates
- Run tests longer to tighten outcome ranges and learn not just which variant wins but how big the win likely is.
- Drew recommends two weeks as a practical rule to compress sampling noise and avoid early bias from frequent visitors.