The short version
- Access is not the problem. 97.5% of the sites answered a request from ChatGPT’s search bot and did not shut it out in robots.txt. Only 2.1% name any AI bot in robots.txt at all, and most of those shut out the bots that collect training data, not the ones that fetch pages to answer a question.
- The sites that are closed mostly did not choose to be. 1.7% of the sites refused a request that identified itself as ChatGPT’s search bot while serving the same page to a browser. For 396 of those 407, robots.txt said nothing against the bot. A firewall, a host or a security plugin did it.
- llms.txt is common, and mostly automatic. 25.9% of the sites have one. 89.6% of Wix sites and 94.0% of Shopify sites do, against 9.3% on Squarespace. In a sample of 397 files, at least 46% were labelled as generated by an SEO plugin or came from one of those two platforms.
- Where sites differ: saying who they are. 24.8% have no structured data on the front page at all and 15.1% no meta description. By industry, the share with Organization or business markup runs from 71.3% for marketing agencies to 47.9% for architecture firms.
- Agencies that sell AI search do the markup work on their own sites. They are no more open to AI bots than other agencies, because almost everyone already is.
What we checked
The sites are those of businesses listed on Google Maps in 33 large cities: 23 in 18 European countries and 10 in the US. 4,100 are marketing and SEO agencies; 19,311 are businesses in 21 other industries, from law firms and dental clinics to wedding venues and logistics. We kept every site whose front page answered normally, 23,411 in all. 16,272 are in Europe and 7,139 in the US. The checks ran in October 2026. We name no site here.
For each site we read what an AI assistant’s crawler would read, without running JavaScript. When Vercel measured AI crawlers at the end of 2024, none of the major ones rendered it.
- the front page as served: its text, title, meta description and the schema.org types in its JSON-LD;
- robots.txt, the Robots Exclusion Protocol file, and which AI bots it shuts out of the whole site;
/llms.txt(the proposal), counted only when it answers with text rather than an HTML page;- a sitemap, from robots.txt or at
/sitemap.xml; - the front page once more, asked for with the user agent of OAI-SearchBot, the bot OpenAI uses for ChatGPT’s search.
The sample is not the whole web. These are businesses with a Maps listing and a website in big cities, and the front page is only one page. The aggregate figures by industry, country and site platform are in a CSV file for anyone who wants to work with them.
Access: almost everyone lets AI in
| What the site does | Sites | Share |
|---|---|---|
| Names at least one AI bot, or every bot, as shut out in robots.txt | 491 | 2.1% |
| …of which only training bots, answering bots allowed | 298 | 1.3% |
| Shuts out every bot in robots.txt | 112 | 0.5% |
| Refuses a request as ChatGPT’s search bot, serves a browser | 407 | 1.7% |
| Closed to ChatGPT’s search by either | 589 | 2.5% |
AI companies run two kinds of bots, and each documents its own: OpenAI, Anthropic, Google, Perplexity, Apple and Common Crawl. Training bots, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, collect pages for building models. Search and user bots, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot, fetch pages to answer a question someone is asking now, and those are the pages an answer can cite. Shutting out the first kind keeps a site out of training data. Shutting out the second keeps it out of the answers: OpenAI says plainly that sites opted out of OAI-SearchBot will not be shown in ChatGPT’s search answers.
The sites that name bots in robots.txt mostly made that distinction. Here is how often each bot is shut out by name:
| Bot | Kind | Shut out by name |
|---|---|---|
| CCBot (Common Crawl) | Training | 1.20% |
| GPTBot (OpenAI) | Training | 1.17% |
| Google-Extended | Training | 0.85% |
| ClaudeBot (Anthropic) | Training | 0.79% |
| Applebot-Extended | Training | 0.56% |
| PerplexityBot | Search | 0.24% |
| ChatGPT-User | Search | 0.23% |
| OAI-SearchBot | Search | 0.11% |
| Claude-SearchBot, Claude-User | Search | 0.05% each |
The firewall refusals are a different story. 407 sites served their front page to a browser and refused the same page to a request that called itself OAI-SearchBot, with a 403 or a similar status. For 396 of them, robots.txt did not shut that bot out. The site’s own rules say yes and its server says no, which is rarely anyone’s decision. It is the kind of thing a bot-protection setting at a CDN, a hosting firewall or a WordPress security plugin can do without anyone choosing it. Defaults have moved that way: in July 2025 Cloudflare announced it was changing its default to block AI crawlers.
To make sure these were not one-off errors, we asked 150 of them again, once as a browser and once as the bot, a few seconds apart. 147 refused the bot again while serving the browser. They sat behind a mix of servers and networks: plain nginx and Apache, Cloudflare, Akamai, Sucuri, CloudFront and LiteSpeed among them.
Counted either way, nearly every site in the sample can be read by the bots that put pages into answers. If a business is missing from what ChatGPT or Perplexity say, a locked door is rarely the reason.
llms.txt: one site in four, mostly by default
llms.txt is a plain-text file that tells AI tools what a site is about and where its key pages are, proposed by Jeremy Howard in September 2024. 6,068 of the 23,411 sites have one: 25.9%, and 30.6% in the US against 23.9% in Europe. For a format proposed in 2024, that looks like fast adoption. The platforms tell a different story.
| Site platform | Sites | Have llms.txt |
|---|---|---|
| Shopify | 436 | 94.0% |
| Wix | 550 | 89.6% |
| Next.js | 1,000 | 36.2% |
| Webflow | 1,063 | 30.8% |
| WordPress | 12,644 | 25.1% |
| Framer | 78 | 19.2% |
| Squarespace | 1,270 | 9.3% |
Nine Wix or Shopify sites in ten having the file is not nine owners in ten deciding to write it. Rates that high point to the platform producing it.
To see what is in the files, we fetched 397 of them at random. 18.1% begin by saying they were generated by Yoast SEO, 8.6% by Rank Math and 5.8% by All in One SEO, the three big WordPress SEO plugins, each of which now offers to generate the file ( Yoast, Rank Math). Another 13.8% are on Shopify or Wix. That makes at least 46% made by a plugin or a platform. The rest we cannot attribute: many are clearly written for the site, some read like a generated summary without saying so.
A generated file is not worthless. It usually lists the site’s main pages, which is the point of the format. But it does mean that “has an llms.txt” says little about whether anyone at the business has thought about what an AI tool reads, and it is not a sign of being ahead. Whether an assistant reads the file at all when it answers is a separate question, and not one this study measures.
Where sites differ: saying who they are
When an assistant reads a business’s site, the plainest help is a page that states what the business is, what it does and where. Two cheap signals of that are a meta description and schema.org markup that names the business as an Organization or one of the business types (LocalBusiness, Dentist, LegalService and the like), which Google documents for its own use. Across all sites:
- 61.7% have Organization or business markup on the front page;
- 24.8% have no structured data on the front page at all;
- 15.1% have no meta description;
- 7.2% have FAQ markup;
- 3.0% show fewer than 300 characters of text without JavaScript, so a crawler that does not run scripts sees an almost empty page;
- 91.8% have a sitemap.
By industry, the differences are much larger than in bot access, which stays between 1.6% and 3.9% closed everywhere:
| Industry | Sites | Organization or business markup | FAQ markup | llms.txt | No meta description | Closed to ChatGPT’s search |
|---|---|---|---|---|---|---|
| Marketing and SEO agencies | 4,100 | 71.3% | 15.1% | 33.1% | 9.8% | 2.3% |
| Software development | 902 | 70.4% | 13.1% | 43.7% | 7.2% | 2.3% |
| Aesthetic clinics | 1,363 | 70.2% | 5.3% | 31.9% | 13.1% | 2.1% |
| IT consulting | 495 | 69.3% | 9.9% | 31.9% | 10.1% | 1.8% |
| Law firms | 1,168 | 69.3% | 6.5% | 27.1% | 11.7% | 2.7% |
| Accounting firms | 567 | 68.1% | 5.8% | 27.5% | 8.6% | 3.4% |
| Solar installers | 480 | 67.1% | 10.2% | 30.4% | 11.0% | 2.7% |
| Translation agencies | 519 | 65.9% | 9.1% | 26.2% | 9.4% | 2.7% |
| Dental clinics | 1,492 | 64.3% | 7.3% | 24.1% | 16.0% | 1.7% |
| Event companies | 763 | 62.6% | 7.3% | 22.8% | 12.3% | 2.0% |
| Coworking spaces | 744 | 58.7% | 4.0% | 21.0% | 15.6% | 2.0% |
| Interior design | 419 | 57.8% | 2.1% | 19.1% | 18.4% | 1.9% |
| Kitchen makers | 162 | 57.4% | 0.6% | 26.5% | 17.3% | 2.5% |
| Private clinics | 1,250 | 56.6% | 5.3% | 20.9% | 20.1% | 2.0% |
| Furniture | 1,657 | 56.0% | 2.8% | 30.8% | 15.1% | 2.7% |
| Recruitment agencies | 1,175 | 55.6% | 4.3% | 19.7% | 13.2% | 3.4% |
| Construction | 643 | 55.1% | 4.2% | 20.4% | 20.4% | 2.2% |
| Logistics | 626 | 54.3% | 7.5% | 20.9% | 20.6% | 3.8% |
| Real estate agencies | 1,137 | 54.2% | 4.0% | 22.3% | 19.1% | 3.9% |
| Language schools | 1,110 | 53.0% | 5.4% | 20.4% | 19.2% | 3.6% |
| Wedding venues | 1,608 | 51.2% | 3.3% | 18.0% | 22.5% | 1.6% |
| Architecture firms | 1,031 | 47.9% | 1.3% | 11.8% | 27.9% | 3.1% |
The pattern is plain. Businesses whose sites are built by people who work on the web, agencies, software firms, IT consultancies, do the basics most often. Businesses that sell something physical or local, venues, architects, builders, language schools, do them least. More than a quarter of architecture firms’ front pages have no meta description.
Between countries the differences are small. The share with business markup runs from 46.8% in Czechia to 68.6% in Spain, and every country is between 1.1% and 4.8% closed to ChatGPT’s search.
Agencies that sell AI search, on their own sites
898 of the 4,100 agencies mention AI search on their front page: GEO, AI visibility, answer engines, AI Overviews or similar. We compared their own sites with those of the other 3,202 agencies.
| Agency site | Mentions AI search | Does not |
|---|---|---|
| Has llms.txt | 55.1% | 26.9% |
| Organization or business markup | 86.4% | 67.1% |
| FAQ markup | 31.2% | 10.6% |
| No meta description | 3.5% | 11.5% |
| Fewer than 300 characters without JavaScript | 0.3% | 5.0% |
| Closed to ChatGPT’s search | 2.1% | 2.4% |
The agencies that sell the work do it on their own sites: twice the llms.txt, three times the FAQ markup, far fewer pages without a description. On access they are the same as everyone else, because almost everyone is already open: there was nothing there to be ahead of.
What to check on your own site
If the figures above hold for your sector, the order of checks is roughly this.
- Ask your server as the bot. Request your front page as OAI-SearchBot, ChatGPT-User and PerplexityBot, and compare with a browser. Our free AI crawler check does this along with robots.txt and llms.txt, and stores nothing. If something is refused that robots.txt allows, look at your CDN’s bot settings and your security plugin before anything else.
- Decide training and search separately. Keeping your pages out of model training does not require keeping them out of answers. If you shut out bots in robots.txt, check you shut out the ones you meant to. Our notes on AI crawlers list which is which.
- Read your llms.txt, if a plugin made one. It is your site’s description to an AI tool. If it lists your cookie policy and none of your services, it is not doing its job.
- Say who you are, in the page. A meta description, Organization or business markup with your name, address and what you do, and a front page whose text is there without JavaScript. None of this is new; a quarter of sites still skip it.
None of these checks tells you whether an assistant actually names you when a buyer asks. A site can be open, marked up and well described and still go unmentioned, because answers are built from much more than one site. The only way to know is to ask the assistants the questions your buyers ask, more than once, and count. That is what Teddit measures.
What the data does not show
- The sample is businesses with a Google Maps listing and a website in 33 large cities. Small-town businesses, large enterprises and sites with no Maps presence are under-represented or absent.
- We read the front page only. Structured data, text and descriptions on other pages are not counted, and JSON-LD is the only markup we read; microdata and RDFa are not.
- The firewall check used the bot’s name from our address, not the bot itself; see the note above.
- The llms.txt attribution is from a random sample of 397 files, and counts only what a file or its platform makes evident. The true share made by tools is higher than 46%, not lower.
- It is one moment, October 2026. Bot defaults at CDNs and plugins change, and so will these numbers.
If you want to cite the figures, the CSV has every aggregate in this article. Questions or corrections: hello@teddit.io.
Sources
- Overview of OpenAI crawlers, OpenAI: OAI-SearchBot, ChatGPT-User and GPTBot.
- Does Anthropic crawl data from the web, Anthropic: ClaudeBot, Claude-User and Claude-SearchBot.
- Google’s common crawlers, Google: Google-Extended.
- Perplexity crawlers, Perplexity: PerplexityBot and Perplexity-User.
- About Applebot, Apple: Applebot-Extended.
- CCBot, Common Crawl.
- RFC 9309: Robots Exclusion Protocol, IETF.
- The /llms.txt file, Jeremy Howard.
- The rise of the AI crawler, Vercel.
- Content Independence Day: no AI crawl without compensation, Cloudflare.
- Organization structured data and control your snippets in search results, Google Search Central.