All checks / AI Discovery
llms-txt-oversizedllms.txt is small
What this check means
Checks that your llms.txt is small enough to help rather than hurt — under 40KB, about 10,000 tokens.
Why it matters
llms.txt is meant to be an index, not the content itself. In our read-cost benchmark a file under roughly 10,000 tokens made a site about twice as cheap for an AI agent to read; a file over it made the site about nine times more expensive than having no file at all. The detail belongs in llms-full.txt.
How we detect it
We measure the UTF-8 byte length of the file. Over 40KB warns; under 200 bytes is treated as an empty stub. The threshold comes from the read-cost benchmark, not from the spec, which sets no limit.
How to fix it
Keep llms.txt to titles, one-line descriptions and links. Move full page content into llms-full.txt. The AI Read Cost tool at /read-cost/<domain> shows which lines to cut, with a reason for each, and the file it would leave.
Whether the file is worth having at all is a fair question. The evidence, for and against, is on Does llms.txt work?
Check whether this applies to your site.
Run an AI audit