Your blog ranks on page one of Google, but ChatGPT has never heard of you. You check your server logs and find the answer: GPTBot, ClaudeBot, and PerplexityBot are hitting your domain and getting a 403 response from your content delivery network before they ever reach your pages. An AI crawler is an automated fetcher — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and similar agents — that AI answer engines use to read and later cite your content. When a CDN's bot-management layer treats these fetchers the same way it treats a scraper, your best pages become invisible to the AI Overviews, Perplexity answers, and Claude conversations that increasingly replace a Google click. AEO Excellence sees this exact failure mode on new client audits more than almost any other technical gap: the robots.txt file is correct, the schema is clean, and the site is still uncited, because a CDN rule upstream of the application is silently returning a challenge page to the crawler before robots.txt is ever read.
Why Does Your CDN Block AI Crawlers?
Your CDN blocks AI crawlers because its default bot-management rules cannot tell the difference between a legitimate AI fetcher and a scraper harvesting your prices or your email addresses. Features like a CDN's bot-fight mode, a managed WAF ruleset, or a JavaScript challenge exist to stop unwanted automated traffic, and by default they treat any fetcher without a browser fingerprint — including GPTBot and ClaudeBot — as suspicious. This is a separate mechanism from robots.txt: the Robots Exclusion Protocol, formalized as RFC 9309, only tells a compliant crawler which paths to skip. It says nothing about firewall-layer blocking, so a fully compliant crawler can still be stopped by your CDN even when your robots.txt allows it outright. The block happens before your application code ever runs, so nothing in WordPress, PHP, or Next.js can fix it — the fix lives entirely in the CDN's dashboard.
How Do You Know an AI Crawler Was Blocked?
You know a crawler was blocked when your CDN's edge logs show a 403 or a challenge-page response on a request whose user agent matches a known AI fetcher, even though the same path returns 200 for a normal browser. Checking this takes four steps:
- Open your CDN's activity or security-event log and filter by user agent for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
- Note the HTTP status code returned to each request — 200 means it reached your server, 403 or a challenge redirect means the CDN stopped it.
- Request the same URL with curl, setting the user agent to the exact crawler string, and compare the response to a normal browser request.
- Cross-check any blocked crawler against your robots.txt file to confirm the block is happening at the CDN, not in your own crawl directives.
The Three Places a CDN Can Silently Block a Bot
A CDN blocks traffic in three separate layers, and each one needs its own allowlist entry: the bot-management engine that scores traffic by behavior, the WAF managed ruleset that blocks by signature, and the rate limiter that blocks by request volume. Allowing a crawler in one layer without the other two still leaves it blocked, which is why a single bot-management toggle often does not fully solve the problem. A crawler that passes the bot-management score can still trip the rate limiter on a large site, and a crawler that clears the rate limiter can still match a WAF signature written for a different kind of bot entirely. Caching makes this worse: if your CDN served a challenge page to GPTBot and cached that response by path rather than by user agent, the next real visitor — a person, not a bot — can receive the same challenge page from cache until the entry expires or you purge it. Always purge the specific paths after changing a bot rule, and check whether your cache configuration varies by user agent on any page a crawler previously hit.
Which AI Crawlers Should Bypass Your CDN Rules?
The AI crawlers that should bypass your CDN's bot-management rules are the ones the major answer engines publish and verify: GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot and anthropic-ai for Claude, PerplexityBot for Perplexity, and Google-Extended for Gemini and AI Overviews. The table below lists what each one feeds and what to allow.
| Crawler user agent | Engine it feeds | What to allow |
|---|---|---|
| GPTBot | ChatGPT browsing and training | Full crawl access, no rate limit below normal traffic levels |
| ClaudeBot / anthropic-ai | Claude | Full crawl access; exclude only pages you would also block from Google |
| PerplexityBot | Perplexity answers | Full crawl access; Perplexity re-crawls trending topics more often than static pages |
| Google-Extended | Gemini and AI Overviews training | Separate from Googlebot — must be allowed on its own line |
Cloudflare, one of the most common CDNs, ships a Verified Bots list inside its bot-management product that already recognizes major AI crawlers by default. The block usually is not that feature — it is Bot Fight Mode or a WAF managed ruleset layered on top of it, and either one needs its own manual exception even after Verified Bots is switched on. Every CDN structures this differently, which is exactly why checking the actual edge logs beats assuming a default setting has you covered.
How Long Until AI Crawlers See the Fix?
An AI crawler sees the fix as soon as it next visits your site, not the moment you save the CDN rule. The rule itself usually propagates to the edge within minutes, but that only clears the door — it does not force a re-crawl. GPTBot typically revisits a mid-size site every 2 to 7 days, so a fix made today usually shows up in ChatGPT's index within that window, while Perplexity refreshes more aggressively for trending topics and can pick up a change sooner. Track the fix by watching your edge logs for the first 200 response from each crawler, not by guessing at a timeline.
Checking three layers across every CDN, confirming the exact user-agent strings each answer engine currently uses, and re-testing after every rule change is manageable for one crawler on one site — it turns tedious fast once you are doing it across multiple properties or a WAF with dozens of managed rulesets. This is the exact gap AEO Excellence's audit stage is built to close: a full crawler-access check across your CDN, WAF, and rate limiter, with a corrected ruleset delivered instead of a bug report. See what a full technical pass covers on our AEO FAQ page.
Won't Allowing More Bots Increase My Security Risk?
Allowing more bots does raise your exposure, but only if you allow the wrong ones. GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are all published, IP-verifiable crawlers from companies with a legal and reputational reason to identify themselves honestly — verify their IP ranges or reverse-DNS before allowlisting the user-agent string, since a scraper can fake the string but cannot fake the source IP. The real question is not whether to allow AI crawlers, it is whether you verified that traffic claiming to be an AI crawler is actually coming from one. A rule that allowlists by user agent alone, with no IP check, is a real vulnerability. A rule that allowlists a verified IP range plus the matching user agent is not.
What to Do Next
Pull your CDN's edge logs for the last seven days, filter for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, and check the status code each one received. If any of them show a 403, a challenge page, or a rate-limit block, that is a lost citation you can fix today. For more on the underlying signals AI crawlers use to decide whether to cite a page once they can reach it, see our AEO blog. If you want a second set of eyes on the full stack — CDN, WAF, rate limiter, and robots.txt together — contact AEO Excellence for a technical crawler-access review and we will tell you exactly which layer is blocking you and how to fix it.