AI crawler access
An assistant can only answer from pages its crawler may read. Teddit reads your robots.txt every night and shows, for each AI crawler, whether it may read your site and which line of the file decides.
Where to find it
At the bottom of Pages, under AI crawler access: one row per AI crawler, with its access and the line of robots.txt that decides it.
A new site is checked as soon as it is added, then every night at 04:30 UTC. Check now reads the files again at once (Member role and above). Through MCP: get_crawler_access.
The crawlers
Each operator runs several crawlers, and they do different things. Blocking one does not block the others.
| Crawler | Operator | Kind | What it does |
|---|---|---|---|
OAI-SearchBot | OpenAI | Search index | Keeps the index ChatGPT search answers from. |
ChatGPT-User | OpenAI | Fetches for a user | Opens a page when someone asks ChatGPT about it. |
GPTBot | OpenAI | Model training | Collects pages to train OpenAI’s models. |
Claude-SearchBot | Anthropic | Search index | Keeps the index Claude’s web search answers from. |
Claude-User | Anthropic | Fetches for a user | Opens a page when someone asks Claude about it. |
ClaudeBot | Anthropic | Model training | Collects pages to train Anthropic’s models. |
PerplexityBot | Perplexity | Search index | Keeps the index Perplexity answers from. |
Perplexity-User | Perplexity | Fetches for a user | Opens a page when someone asks Perplexity about it. |
Googlebot | Search index | Google Search’s crawler; Gemini and AI Overviews answer from its index. | |
Google-Extended | Model training | Decides whether Google may use the pages to train Gemini. | |
Applebot-Extended | Apple | Model training | Decides whether Apple may use the pages to train its models. |
Meta-ExternalAgent | Meta | Model training | Collects pages to train Meta’s models. |
CCBot | Common Crawl | Model training | Builds the open web archive many models are trained on. |
- Search index crawlers keep the index an assistant searches when it answers. Blocking them takes your pages out of those answers.
- Fetches for a user open one page because a person asked the assistant about it.
- Model training decides whether your pages may be used to train future models. It does not change what an assistant finds when it searches today.
How access is decided
Teddit reads the file the way the robots.txt standard (RFC 9309) says crawlers must:
- The rules written for a crawler by name apply to it; the
User-agent: *group applies only to a crawler no group names. - No robots.txt at all (the site answers 404) means every crawler may read everything.
- A robots.txt that cannot be read (no answer, or a server error) means access is Unknown.
| Access | Meaning |
|---|---|
| Allowed | No rule closes anything to this crawler. |
| Some paths closed | The home page is open, but at least one rule closes some addresses (often /wp-admin/). The line shown is the first such rule. |
| Blocked | The home page is closed to this crawler. The line shown is the rule that closes it. |
| Unknown | robots.txt could not be read at the last check. |
User-agent: GPTBot Disallow: / # GPTBot: Blocked, line 2 User-agent: * Disallow: /wp-admin/ # every other crawler: Some paths closed, line 5
When the access of a crawler changed since the check before, the table says what it was (“was allowed”). Every check is kept, so the history of your file is there to read.
llms.txt
Teddit also looks for /llms.txt, a plain-text summary of a site some AI tools read. It is reported as found when the address answers with text; a site that answers every address with its HTML page does not count as having one.
TedditBot when it reads these two files. It reads nothing else on your site for this check.