GPTBot vs OAI-SearchBot: which AI crawlers to block in robots.txt

Two decisions, not one

Robots.txt answers two separate questions for an AI company, and each company asks them with different crawlers:

• May my pages be used to train your models? That is GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and, indirectly, CCBot. Saying no keeps future content out of training data. It does not change what an assistant finds when it searches today.
• May your assistant find and cite my pages when someone asks? That is OAI-SearchBot, Claude-SearchBot and PerplexityBot, plus the fetchers that open one page for one person (ChatGPT-User, Claude-User, Perplexity-User). Saying no to the search crawlers takes the site out of the answers.

Most businesses want to be in the answers, because that is where buyers now ask for recommendations. Whether to give pages for training is a choice each site makes for itself. The two can be set independently, and OpenAI says so in as many words: each setting is independent of the others.

OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User

GPTBot is OpenAI’s training crawler. Disallowing it, in OpenAI’s words, indicates that a site’s content should not be used in training generative AI foundation models (OpenAI).

OAI-SearchBot is the one that matters for ChatGPT’s answers. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and it recommends allowing the bot. A change to robots.txt takes about 24 hours to reach OpenAI’s systems.

ChatGPT-User is not a crawler in the usual sense. It fetches pages for actions a user starts in ChatGPT or a custom GPT, does not crawl the web automatically, and because a person asked, robots.txt rules may not apply to it. OpenAI also says it does not use ChatGPT-User to decide whether content may appear in search: that is OAI-SearchBot’s rule.

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

Anthropic documents three agents on one page (Anthropic):

• ClaudeBot collects web content that may be used for training. Restricting it signals that the site’s future materials should be excluded from training datasets.
• Claude-SearchBot crawls to improve the quality of search results for Claude’s users. Disabling it prevents your content from being indexed for search, which Anthropic says may reduce your site’s visibility in user search results.
• Claude-User opens pages when a person asks Claude a question. Disabling it prevents Claude from retrieving your content in response to a user query.

Anthropic says its bots honour robots.txt and the non-standard Crawl-delay extension, and that blocking its IP addresses may not work correctly or guarantee an opt-out, so robots.txt is the way to do it. The opt-out has to be set on each subdomain as well.

Perplexity: PerplexityBot and Perplexity-User

PerplexityBot surfaces and links websites in Perplexity’s search results, and Perplexity states it is not used to crawl content for AI foundation models (Perplexity). There is no training reason to block it: blocking it only removes the site from the place where Perplexity looks for answers. Perplexity recommends allowing it in robots.txt and allowing requests from its published IP ranges.

Perplexity-User fetches a page when a user asks a question and may link to it in the answer. Since a user requested the fetch, Perplexity says it generally ignores robots.txt rules. It is not used for training either.

Google and Apple: the “-Extended” tokens

Google-Extended is not a crawler at all. It has no user agent of its own: Google crawls with its usual agents, and the token in robots.txt only controls use. It decides whether content may be used to train the Gemini models and for grounding in Gemini Apps and on Vertex AI. Google states it does not affect a site’s inclusion in Google Search and is not a ranking signal (Google).

What Google-Extended does not control is AI Overviews and AI Mode. They are part of Search, and Google says the control for them is the ordinary one: robots.txt rules for Googlebot, and the nosnippet, data-nosnippet, max-snippet or noindex controls to limit what is shown. There are no additional technical requirements to appear in them (Google Search Central). So there is no way to stay in Google Search but out of AI Overviews with robots.txt alone, and blocking Googlebot to avoid AI Overviews removes the site from Google Search.

Applebot-Extended works the same way. It does not crawl pages; it decides whether what Applebot collected may be used to train Apple’s foundation models. Pages that disallow it can still be included in search results in Spotlight, Siri and Safari, and the rule is not used for ranking (Apple).

CCBot builds Common Crawl’s open archive of the web (Common Crawl). Common Crawl does not describe it as an AI crawler, but the archive is open for anyone to use, and in our study CCBot was the bot sites shut out by name most often.

Robots.txt to copy

In the answers, out of training. The setup most businesses that care about training want. Search and user crawlers are not named, so the * group lets them in; the training tokens are closed:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

Several User-agent lines can share one group of rules, as the robots.txt standard allows (RFC 9309).

Open to everything. If you are happy for your pages to be used for training too, you need no AI-specific lines at all. A plain file is enough:

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Out of AI assistants, still in Google Search. Possible, at a price: the site disappears from ChatGPT, Claude and Perplexity answers. Google’s AI Overviews and AI Mode cannot be closed this way, as they follow Googlebot.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

The user fetchers (ChatGPT-User, Perplexity-User) may not follow the file, as their operators say, so even this does not guarantee that a page is never opened for a user.

Mistakes that close the door by accident

• A named group replaces the general one. Under the standard, a crawler that finds a group with its own name obeys only that group and skips the * group. If you add User-agent: OAI-SearchBot with a single rule, your other rules no longer apply to it. Repeat in the named group whatever should still hold.
• A robots.txt that errors. When the file answers with a 5xx server error, the standard tells crawlers to assume everything is disallowed. A 404 means the opposite: everything is allowed (RFC 9309).
• The firewall says no when robots.txt says yes. Bot protection at the CDN or the host can refuse agents it does not know. In our check of 23,411 business websites, 2.5% were closed to ChatGPT’s search, and 407 sites refused a request that called itself OAI-SearchBot while serving the same page to a browser. For 396 of them, robots.txt allowed the bot.
• Blocking the search bot to stop training. In the same study, sites named the training bots far more often than the search bots: GPTBot was shut out by name on 1.17% of sites, OAI-SearchBot on 0.11%. Most sites that name bots at all make the distinction; check that yours does.
• Forgetting subdomains. Robots.txt applies to one host. A blog on blog.example.com or docs on docs.example.com need their own file.

Check what your site actually says

Our free AI crawler check reads your robots.txt the way the standard says crawlers must, shows the verdict for each crawler above with the line that decides it, and requests your front page as OAI-SearchBot, Claude-SearchBot and PerplexityBot to catch a firewall that refuses them. It needs no account and stores nothing.

Inside Teddit, every tracked site’s robots.txt is read every night, and when a crawler’s access changes, the table says what it was (AI crawler access). Access is only the door, though: whether an assistant then names you is a separate question, which how an assistant decides which brands to name goes into.

Sources

• Overview of OpenAI crawlers, OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User.
• Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic: ClaudeBot, Claude-SearchBot and Claude-User.
• Perplexity crawlers, Perplexity: PerplexityBot and Perplexity-User.
• Google’s common crawlers, Google: Google-Extended.
• AI features and your website, Google Search Central: AI Overviews and AI Mode.
• About Applebot, Apple: Applebot and Applebot-Extended.
• CCBot, Common Crawl.
• RFC 9309: Robots Exclusion Protocol, IETF.