Which AI Crawlers to Allow in Your robots.txt File

Your site can rank on page one of Google and still be invisible in ChatGPT. The reason is almost always the same: robots.txt. A default file from a CMS install, a security plugin's blanket Disallow rule, or a WAF that blocks unrecognized user agents will silently keep every AI crawler out, and you will never see an error for it — the bot just stops visiting.

An AI crawler allowlist is the set of User-agent directives in your robots.txt file that explicitly permit bots like GPTBot and ClaudeBot to read your pages and cite them in an AI-generated answer. Get the allowlist wrong and the content itself does not matter, because the crawler never sees it.

What Counts as an AI Crawler?

An AI crawler is any automated bot that fetches your pages to feed a large language model, either for long-term training or for a live, per-query answer. That splits into two behaviors: training crawlers visit periodically and store content for a future model version, while retrieval crawlers like PerplexityBot fetch a page in real time while answering a specific user question. Both need to be allowed if you want to be cited, because a blocked retrieval crawler cannot answer with your content even once.

Which AI Crawlers Should You Allow in robots.txt?

At minimum, allow GPTBot, ClaudeBot, Google-Extended, PerplexityBot, anthropic-ai, CCBot, and Applebot-Extended. Each one maps to a different AI product your customers are already using to research your category.

The Google-Extended user agent and Googlebot are not the same directive and control different things: Googlebot governs whether your pages are indexed for standard Google Search, while Google-Extended governs only whether Google may use your content to train Gemini and generate AI Overviews. Disallowing Google-Extended does not remove you from Google Search, and allowing it does not change your search ranking.

What Happens If You Block the Wrong Bot?

A blocked crawler does not retry, complain, or send you a notification — it simply skips your domain on its next crawl cycle and moves on to a competitor that did allow it. GPTBot revisits a typical mid-size site every 2 to 7 days, so a misconfigured robots.txt does not cost you one missed answer, it costs you every answer in that window, compounding for as long as the block stays in place.

This is the gap an AEO audit exists to close: reading your live robots.txt line by line against the current AI crawler list, because the list changes as new engines launch and old defaults quietly go stale. If you want that check run against your actual file rather than guessed at, see our AEO FAQ for the full crawler-by-crawler reference we maintain.

How Do You Add These Crawlers to robots.txt?

Adding an AI crawler allowlist is a five-step edit to a plain text file at your domain root, and it takes effect the next time each bot revisits.

1. Open or create robots.txt at the root of your domain, e.g. yoursite.com/robots.txt.
2. Add one User-agent block per crawler you want to allow, each followed by Allow: /.
3. Keep any existing Disallow rules for admin paths, checkout flows, or staging directories in their own separate blocks, not mixed into the AI crawler blocks.
4. Save the file and load yoursite.com/robots.txt directly in a browser to confirm it returns plain text with no server error.
5. Re-check in 30 days by asking your top 10 category questions in ChatGPT, Perplexity, and Claude, and logging which sites get cited.

Should You Ever Block an AI Crawler?

Yes, but only deliberately, not by default. The standard reason to block is training-data avoidance: if you do not want a specific engine training on your content, disallowing GPTBot, ClaudeBot, or Google-Extended does stop training-focused crawling from that engine, because the major commercial crawlers publicly commit to honoring robots.txt. What blocking does not do is opt you out of citation once your content is already in an index, and it does not stop smaller or scraping-focused bots that ignore the file entirely — those need a WAF or rate limiter, not a robots.txt line.

There is also a middle option worth naming: allowing a crawler for retrieval while blocking it for training is not possible with one robots.txt line today, because most engines publish a single bot that covers both uses. Anthropic is the exception — anthropic-ai handles retrieval and ClaudeBot handles training as separate user agents — but for GPTBot, PerplexityBot, and most others, one allow-or-block decision covers everything that bot does. That is why the block decision should be made deliberately, not inherited from whatever a plugin shipped with by default.

How Do You Confirm the Crawlers Are Actually Visiting?

Editing robots.txt only sets a permission; it does not prove a crawler read it. Confirm real visits by pulling your web server's access log and searching it for each bot's user-agent string, since every major AI crawler identifies itself by name on every request.

1. Open your server's raw access log, usually available as access.log in your hosting control panel's log viewer.
2. Search it for GPTBot, ClaudeBot, Google-Extended, and PerplexityBot as literal strings.
3. Note which pages each bot actually fetched, not just that it visited — a bot that only hit your homepage never read the page you want cited.
4. If a crawler you allowed shows zero hits after two full weeks, check for a CDN, WAF, or bot-management layer sitting in front of your server, since those can block a user agent independently of robots.txt.

That last check matters because robots.txt compliance and infrastructure-level blocking are two separate systems. A site can carry a perfect allowlist in robots.txt and still be invisible to GPTBot because a security service in front of the server rejects the same user agent before the request ever reaches robots.txt.

Do I Need to Rebuild My Whole SEO Setup for This?

No, and this is the objection worth naming directly: teams hear "AI crawler allowlist" and assume it is a large technical project that competes with existing SEO work for engineering time. It is not. Editing robots.txt is a single text file change with no build step, no deploy pipeline, and no risk to your existing Google ranking, because the crawler directives that improve AEO (clean allowlisting, valid schema, question-based headings) are additive to SEO rather than in conflict with it.

Where teams do get stuck is knowing which of the 10-plus AI crawlers currently matter enough to bother with, and confirming their existing robots.txt has not already blocked half of them through an old plugin default. That is a fifteen-minute conversation, not a project.

What Should You Do Next?

Pull up your live robots.txt right now and check it against the table above. If GPTBot, ClaudeBot, or Google-Extended is missing or explicitly disallowed, that is lost citation opportunity you can fix today. If you want a second pair of eyes on the full file, along with the rest of your AEO foundation, reach out and we will walk through your specific robots.txt together.