Can ChatGPT, Claude and Perplexity read your site?
Enter a website. We read its robots.txt for 13 AI crawlers, ask its front page as a browser and as the assistants’ search crawlers, and show what a crawler that runs no JavaScript finds there. Nothing is stored.
The AI crawlers this check reads
Each operator publishes its crawlers’ names. They do different jobs, and closing each one has a different effect.
| Crawler | Operator | Job | What it does |
|---|---|---|---|
OAI-SearchBot | OpenAI | Search index | Keeps the index ChatGPT search answers from. |
ChatGPT-User | OpenAI | Fetches for a user | Opens a page when someone asks ChatGPT about it. |
GPTBot | OpenAI | Model training | Collects pages to train OpenAI’s models. |
Claude-SearchBot | Anthropic | Search index | Keeps the index Claude’s web search answers from. |
Claude-User | Anthropic | Fetches for a user | Opens a page when someone asks Claude about it. |
ClaudeBot | Anthropic | Model training | Collects pages to train Anthropic’s models. |
PerplexityBot | Perplexity | Search index | Keeps the index Perplexity answers from. |
Perplexity-User | Perplexity | Fetches for a user | Opens a page when someone asks Perplexity about it. |
Googlebot | Search index | Google Search’s crawler; Gemini and AI Overviews answer from its index. | |
Google-Extended | Model training | Decides whether Google may use the pages to train Gemini. | |
Applebot-Extended | Apple | Model training | Decides whether Apple may use the pages to train its models. |
Meta-ExternalAgent | Meta | Model training | Collects pages to train Meta’s models. |
CCBot | Common Crawl | Model training | Builds the open web archive many models are trained on. |
Questions
What is the difference between OAI-SearchBot and GPTBot?
OAI-SearchBot keeps the index ChatGPT search answers from; GPTBot collects pages to train OpenAI’s models. Blocking GPTBot keeps your pages out of training and does not remove you from ChatGPT search; blocking OAI-SearchBot does.
Does blocking Google-Extended remove a site from AI Overviews?
No. Google-Extended only decides whether Google may use the pages to train Gemini. AI Overviews and AI Mode answer from Google Search’s index, which Googlebot keeps.
My robots.txt allows the crawlers, yet the check shows a refusal. Why?
A firewall or a bot-protection service can turn away requests that name an AI crawler, whatever robots.txt says. Real crawlers come from their operators’ published addresses, which this check cannot use, so a refusal here is a sign to look at the firewall settings, not proof.
Why does text without JavaScript matter?
Many AI crawlers read the HTML as the server sends it and run no JavaScript. A page that only fills in after scripts run can look nearly empty to them.
Does llms.txt make assistants recommend a site?
There is no published evidence that the assistants use llms.txt to choose what they recommend. The check shows whether you have one; it is a small signal at most.
Teddit runs this check every day for the sites it tracks; see AI crawlers in the docs. To see what the assistants say about you, try the free visibility check or see what tracking costs.