llms.txt: what it is, and whether AI assistants actually read it

What llms.txt is

llms.txt was proposed by Jeremy Howard in September 2024. It is a plain Markdown file served at /llms.txt, meant for language models that read a site when someone asks them about it: a short description of the site and a list of its key pages, so a model does not have to work through menus, scripts and cookie banners to find them.

The format is small. Only the first line is required:

• A title (# Name): the site or project.
• A summary in a quote block (>): what it is, in a sentence or two.
• Any notes in plain paragraphs: what a reader should know first.
• Sections of links under ## Headings, one [name](url): note per line. A section called ## Optional holds what can be skipped.

Ours, shortened:

# Teddit

> Teddit (teddit.io) is AI visibility tracking: it asks ChatGPT, Claude,
> Perplexity, Gemini and Google's AI answers the questions your customers ask,
> and shows who gets named, who gets linked and what changed.

Teddit is not related to the open-source Reddit front-end also called Teddit.

## Site

- [Pricing](https://teddit.io/pricing): pay per answer, no subscription
- [AI crawler check](https://teddit.io/tools/ai-crawler-check): free check of robots.txt for AI crawlers

## Optional

- [Terms of Service](https://teddit.io/terms)

The proposal also suggests a Markdown copy of each page at the same address with .md added. llms-full.txt, the whole site or documentation in one file, is not in the proposal; it is a convention that grew up around it, mostly on documentation sites.

Who asked for ours: 18 days of server logs

teddit.io serves an llms.txt built with the site: a summary, then every page with what it is about. Our server logs every request from a crawler (not from people). From 16 October to 4 November 2026 we counted what each AI company’s crawler asked for: everything, then robots.txt, the sitemap and llms.txt.

Four requests in 8,490: GPTBot once on 16 October, PerplexityBot on 30 October, GoogleOther and ClaudeBot on 1 November. Each got the file. None came back for it. In the same days the same crawlers asked for robots.txt 480 times and for the sitemap 255 times, and read 844 different addresses.

The crawlers that fetch pages while an assistant is answering someone (OAI-SearchBot, ChatGPT-User, Perplexity-User, Claude-User) made 1,127 requests between them and never asked for llms.txt. The four that did are crawlers that collect pages in bulk, and they treated it as one more page.

What the companies say

Google is the only one that says it outright. Its guide to Google’s AI features lists llms.txt among the things you do not need: Google Search does not use these files, and having one neither helps nor harms a site in Google Search, AI Overviews and AI Mode included. Google added that wording in June 2026 (Search Central updates), saying such files are fine to keep for other systems that use them.

OpenAI, Anthropic and Perplexity do not mention llms.txt in what they publish about their crawlers (OpenAI, Anthropic, Perplexity). They describe robots.txt, and nothing else, as the way a site tells them what to read.

Yet all three publish an llms.txt of their own, on their developer documentation (OpenAI, Anthropic, Perplexity). That is where the format works: a developer points a coding assistant at the docs, and the file gives it the map. It is a file for an AI tool someone has sent to your site, not a signal that makes an assistant choose your site.

Who has one

In our study of 23,411 business websites in Europe and the US, 25.9% had an llms.txt. Most did not choose to: 94.0% of Shopify sites and 89.6% of Wix sites had one, against 9.3% on Squarespace, and in a sample of 397 files at least 46% were labelled as made by an SEO plugin or the platform.

A file made by default often lists the privacy policy and the cart and leaves out what the business sells. It is harmless, but it is not a description of the business anyone has written.

Should you have one?

Yes, if it is cheap for you, and with modest hopes. It costs one file and an hour, it harms nothing, and it is useful to the AI tools that do read it: coding assistants, research agents and anyone who points a model at your site.

Do not expect it to get you named. Nothing we found says ChatGPT, Claude, Perplexity or Google’s AI answers use it to decide which brands to recommend. What they do use is what a page says when their search finds it. Worth doing first:

1. Let the search crawlers in. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot need to be allowed in robots.txt and not refused by your firewall (which crawlers to allow).
2. Put the words in the HTML. A page whose text only appears after its scripts run may be read as empty.
3. Have a page for each thing you are asked about: what you do, for whom, at what price, compared with whom. That is what an assistant quotes.

If you keep an llms.txt, write it yourself: one honest sentence on what you do, then your service, pricing and contact pages, with a plain note on each. If a plugin made yours, open it and check it lists what you sell.

Check yours

Our free llms.txt checker reads a site’s file against the format, opens the links in it, looks for llms-full.txt, and drafts a first file when there is none. The AI crawler check does the part that matters more: whether each AI crawler may read the site at all. Neither needs an account.

Sources

• The /llms.txt file, Jeremy Howard: the proposal and its format.
• Google’s guide to optimizing for generative AI features on Google Search, Google Search Central: llms.txt is not used by Google Search.
• Search Central updates, Google: the June 2026 note on llms.txt files.
• Overview of OpenAI crawlers, OpenAI.
• Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic.
• Perplexity crawlers, Perplexity.
• The llms.txt files of OpenAI’s, Anthropic’s and Perplexity’s developer documentation.
• We checked 23,411 business websites. Almost all of them let AI in, Teddit: how many sites have an llms.txt, and who made it.