Does llms.txt work?

Is llms.txt actually used?

Rarely. Four independent server-log studies in 2025 and 2026 reached the same result from different data: AI crawlers ask for robots.txt hundreds of times for every time they ask for llms.txt, and most of the fetches that do happen come from SEO tools checking whether the file exists. Meta's crawler is the one documented exception.

Publishing is rare too. Our registry has probed 236,728 domains; 16,814 publish a file. Over the 235,926 of those drawn from a ranked list of the web, adoption is 6.9%. More of those domains refused our request outright (17,144) than published an llms.txt (16,814). Adoption runs from 27.8% on .ai down to 7.6% on .gr.

Adoption excludes our examples directory and audited domains, both selected because they were likely to have a file. How it is counted.

Does Google, ChatGPT or Claude use llms.txt?

No provider has said in writing that its search product reads the file. Google has said the opposite. The nearest things to a yes are indirect: Anthropic asked its documentation host to publish one, Google's Lighthouse now checks for it under "agentic browsing", and coding agents such as Claude Code appear among the few real fetchers in crawler logs.

That pattern matters. The file was proposed for agents reading documentation, and that is the one place the evidence says it is used. It was never adopted by search, and the search-visibility promise attached to it later has no data behind it.

Is llms.txt worth it?

For citations in AI search: nobody has shown a benefit, and the largest study found none. For an agent reading your site: it depends entirely on the size of the file, which is the thing every other study ignores. Each treats llms.txt as present or absent. We measured what the file does.

That is from 20 domains, five questions each, four retrieval routes, 400 attempts, run 6 September 2026 under benchmark v1.0.0. Read flat, the llms.txt route cost a median of 21,196 tokens to answer against 2,294 for the conventional site, because half the files were bigger than the site they described. Split by size, every small file won and every large file lost. Nothing crossed. llms-full.txt answered 0 of 10 attempts: both files exceeded the token budget outright.

Twenty domains that all begin with "a" is a sample for finding methodology faults, not for generalising. The defects are disclosed on the findings page; phase 2 is 166 matched domains across eleven verticals, selected and not yet run.

The studies, for and against

As published by each source. We link them so you can check the sample and the method.

What we measure that nobody else does

• File size, not just presence. Every other study is binary. The benchmark above is the only size-conditioned result we know of, and size turned out to be the whole story.
• The file's history. The registry keeps every version of every file it finds (17,524 versions so far, 98 changed in the last 30 days), so churn, staleness and abandonment can be measured later. Nobody can backfill that.
• Fetched versus used. Log studies count crawler hits. Our AI Citation Check asks AI search engines real questions and records who they cite, before and after a file is deployed. We will publish the cohort result once enough sites have both readings; we will not publish it early.

Verdict

Publish one if it is small and accurate. It costs minutes, and for any agent that reads it, it halves the cost of understanding your site. Do not publish a large one: past roughly 10,000 tokens it makes your site more expensive to read than having no file. Do not expect it to change your AI search citations. Nobody has shown that it does, and the people saying otherwise are mostly selling generators, including, until this year, us.

The things that do show up in citation data are older and duller: reachable pages, crawlers not blocked, clear answers on the page itself, structured data. That is what the rest of the audit checks.