215: The Agent-Broken Web - Why AI Can't See Your Website

Jan 28, 2026

Ask episode

Chapters

Transcript

Episode notes

Your website might rank #1 on Google but be completely invisible to ChatGPT, Claude, and Perplexity. In this episode, let's break down why a huge chunk of the web is fundamentally broken for AI systems - not because of bad content, but because of technical decisions that made sense for humans but make sites invisible to the AI systems rapidly becoming the front door to the internet.

Chapter Timestamps

00:00:00 - Introduction: The new game your website is losing
00:01:43 - The Scale of the Problem: AI crawler traffic explosion
00:05:19 - The JavaScript Problem: Why AI crawlers can't see your content
00:10:28 - The Bot Protection Paradox: Accidentally blocking AI
00:14:40 - The Speed Requirement: Why 200ms matters
00:17:46 - AI Agents Are Struggling Too: Browser agents and their limitations
00:20:46 - How to Fix It: 6 things you need to do
00:25:33 - Closing: The web is adapting again

Key Statistics

569 million GPTBot requests on Vercel's network in a single month
370 million ClaudeBot requests in the same period
305% growth in GPTBot traffic (May 2024 to May 2025)
157,000% increase in PerplexityBot requests year-over-year
33% of organic search activity now comes from AI agents
~40% failure rate for the best AI browser agents on complex tasks

The 6 Things to Fix

Implement Server-Side Rendering (SSR) - If your site uses a JavaScript framework (React, Vue, Angular) with client-side rendering, switch to SSR or static site generation immediately. Use Next.js, Nuxt, or a pre-rendering service.
Add Structured Data with JSON-LD - Expose key information in machine-readable format using schema.org markup. Microsoft confirmed Bing uses this to help Copilot understand content.
Optimize for Speed - Target server response time under 200ms. First Contentful Paint under 1 second. Largest Contentful Paint under 2.5 seconds.
Check Your Bot Protection Settings - Review Cloudflare, AWS WAF, or your CDN's bot management. Make a deliberate decision about GPTBot, ClaudeBot, and PerplexityBot access.
Kill Infinite Scroll and Lazy Loading for Content - Use paginated URLs with standard HTML links. Ensure high-value content is in the initial HTML response.
Keep Sitemaps Current - Maintain proper redirects, consistent URL patterns, and fix broken links.

Tools Mentioned

Glimpse - Free tool to test how AI sees your website: glimpse.webperformancetools.com

Show Links

Sources Referenced in This Episode

AI Crawler Statistics:

JavaScript Rendering:

Episode URL: http

No Hacks is a podcast about web performance, technical SEO, and the agentic web. Hosted by Slobodan "Sani" Manic.