The Stack Overflow Podcast Why Stack Overflow and Cloudflare launched a pay-per-crawl model
5 snips
Feb 19, 2026 
Will Allen, Cloudflare VP who builds bot management and pay-per-crawl tooling. Josh Zhang, Stack Overflow SRE who runs bot traffic and DDoS defenses. Janice Manningham, Stack Overflow product leader on data licensing and access controls. They discuss why pay-per-crawl was needed, how crawlers and scraping evolved, technical 402 handling and tooling, and programmatic licensing vs enterprise deals.
AI Snips
Chapters
Transcript
Episode notes
Open/Block Model Is Broken
- The old open-or-block web model has broken due to AI crawlers harvesting content at scale.
- Platforms must revisit access policies to protect commercial value while preserving community access.
Bots Evolved Into Sophisticated Scrapers
- Josh described an arms race where bots evolved from denial tactics to sophisticated scraping that mimics normal user traffic.
- Bots now consume ad impressions and value while hiding as legitimate visitors, creating new harms for sites.
Give Publishers Control Over Crawls
- Put publishers in the driver's seat to decide how crawlers access and monetize content.
- Use bot identity and policy controls to allow, block, rate-limit, or charge crawlers based on publisher preference.