# llmstxt.studio > llmstxt.studio runs a free AI Readiness Audit of any website, 41 checks across access, discovery, content understanding, machine-readable data and answer readiness, and keeps a public llms.txt Registry that records what the ranked top million domains publish and how it changes over time. The short index of this site is at https://llmstxt.studio/llms.txt. This file adds the documentation for every audit check. ## How the audit works - Run it on any domain at https://llmstxt.studio/audit/. It is free and needs no account. - Every check reads from one fetch of the site, so the audit costs the same whatever the check count. - A check that does not apply is skipped. A skip is not a failure and not a pass; it is excluded from the score. - A domain that times out, refuses the request or redirects elsewhere is reported as unreachable, never as "no llms.txt". - When a site blocks every AI crawler the overall score is capped and the report says so in full. - Adoption figures are computed over a stated ranked sample only. The live figure is at https://llmstxt.studio/stats. ## Our position on llms.txt Not for AI search, as far as anyone can show: Google Search does not use the file and no major AI provider has confirmed that its search product reads it. For AI agents that read a site directly, a small file helps: on our data, files under about 10,000 tokens made a site roughly twice as cheap to read, and larger files made it more expensive than having none. Every study we could find, for and against, is at https://llmstxt.studio/does-llmstxt-work and the benchmark is at https://llmstxt.studio/research/ai-read-cost. ## AI Access checks ### Homepage responds successfully https://llmstxt.studio/checks/homepage-http-error What: Checks that your homepage returns a successful response to an AI crawler. Why it matters: If the homepage does not load for a crawler, nothing else about your setup matters. It is also the page most systems try first, so a failure here shapes their view of the whole domain. How we detect it: We request your homepage with an identifiable bot user-agent, following redirects, and require a 2xx response. How to fix it: Confirm the homepage loads over HTTPS for a non-browser user-agent. A failure only for bots usually means a firewall or bot-management rule. ### No bot challenge blocks AI systems https://llmstxt.studio/checks/bot-challenge-blocking-agents What: Checks whether your homepage answers an identified bot with a challenge or a refusal rather than your content. Why it matters: This is the quiet one. Your robots.txt can welcome every AI crawler while your CDN turns them away at the door, and nothing in your configuration will tell you — the rules that permit them and the rules that block them live in different systems. How we detect it: We request your homepage with an identifiable bot user-agent and look for 401, 403 or 429, or a Cloudflare-style interstitial (a cf-mitigated header, or a 503 from a Cloudflare edge). How to fix it: Allow-list the AI crawlers you intend to permit in your CDN or WAF bot-management rules, not only in robots.txt. ### Homepage redirects are short https://llmstxt.studio/checks/redirect-chain-excessive What: Checks how many redirects a crawler follows before reaching your homepage. Why it matters: Every hop is a round trip a crawler pays for and another link that can break. One hop — http to https, or apex to www — is normal and expected. Three or more usually means several rules stacked over the years, each unaware of the others. How we detect it: We request your homepage without following redirects automatically and walk the Location headers by hand, recording each hop. Two or more hops warns. How to fix it: Collapse the chain so the first request lands on the canonical URL in at most one hop. ### GPTBot can reach your site https://llmstxt.studio/checks/gptbot-blocked What: Checks whether your robots.txt denies GPTBot, the crawler operated by OpenAI, access to your site. Why it matters: Your pages are left out of the data OpenAI uses to train its models. ChatGPT search and live browsing use different crawlers (OAI-SearchBot and ChatGPT-User), so blocking GPTBot alone does not take you out of ChatGPT answers. How we detect it: We parse robots.txt per RFC 9309 and ask whether GPTBot is permitted to fetch "/". The most specific matching user-agent group wins, so a named GPTBot group replaces any wildcard rules rather than combining with them. A bare "Disallow:" permits everything and is not a block. How to fix it: Remove the blanket "Disallow: /" from the group covering GPTBot, or add an explicit group permitting it: User-agent: GPTBot Allow: / ### OAI-SearchBot can reach your site https://llmstxt.studio/checks/oai-searchbot-blocked What: Checks whether your robots.txt denies OAI-SearchBot, the crawler operated by OpenAI (ChatGPT search), access to your site. Why it matters: ChatGPT search cannot read or link to your pages. This crawler makes the live links that ChatGPT shows in its answers. How we detect it: We parse robots.txt per RFC 9309 and ask whether OAI-SearchBot is permitted to fetch "/". The most specific matching user-agent group wins, so a named OAI-SearchBot group replaces any wildcard rules rather than combining with them. A bare "Disallow:" permits everything and is not a block. How to fix it: Remove the blanket "Disallow: /" from the group covering OAI-SearchBot, or add an explicit group permitting it: User-agent: OAI-SearchBot Allow: / ### ClaudeBot can reach your site https://llmstxt.studio/checks/claudebot-blocked What: Checks whether your robots.txt denies ClaudeBot, the crawler operated by Anthropic, access to your site. Why it matters: Claude cannot read your pages. It then has nothing from your site to use when it answers questions about your products. How we detect it: We parse robots.txt per RFC 9309 and ask whether ClaudeBot is permitted to fetch "/". The most specific matching user-agent group wins, so a named ClaudeBot group replaces any wildcard rules rather than combining with them. A bare "Disallow:" permits everything and is not a block. How to fix it: Remove the blanket "Disallow: /" from the group covering ClaudeBot, or add an explicit group permitting it: User-agent: ClaudeBot Allow: / ### PerplexityBot can reach your site https://llmstxt.studio/checks/perplexitybot-blocked What: Checks whether your robots.txt denies PerplexityBot, the crawler operated by Perplexity, access to your site. Why it matters: Perplexity cannot read or link to your pages. Perplexity shows links to its sources, so your site cannot appear in them. How we detect it: We parse robots.txt per RFC 9309 and ask whether PerplexityBot is permitted to fetch "/". The most specific matching user-agent group wins, so a named PerplexityBot group replaces any wildcard rules rather than combining with them. A bare "Disallow:" permits everything and is not a block. How to fix it: Remove the blanket "Disallow: /" from the group covering PerplexityBot, or add an explicit group permitting it: User-agent: PerplexityBot Allow: / ### Google-Extended can reach your site https://llmstxt.studio/checks/google-extended-blocked What: Checks whether your robots.txt denies Google-Extended, the crawler operated by Google (Gemini), access to your site. Why it matters: Gemini cannot use your content in its answers. This token controls AI use only. It does not change your Google Search rank. How we detect it: We parse robots.txt per RFC 9309 and ask whether Google-Extended is permitted to fetch "/". The most specific matching user-agent group wins, so a named Google-Extended group replaces any wildcard rules rather than combining with them. A bare "Disallow:" permits everything and is not a block. How to fix it: Remove the blanket "Disallow: /" from the group covering Google-Extended, or add an explicit group permitting it: User-agent: Google-Extended Allow: / ### robots.txt is published https://llmstxt.studio/checks/robots-txt-missing What: Checks whether a robots.txt is served at your domain root. Why it matters: Without one, crawlers are permitted by default — so this is not a blocker. But robots.txt is also where you declare your sitemap and where you can explicitly welcome AI crawlers, and its absence means you have no way to express either. How we detect it: We request https://yourdomain/robots.txt and require a 2xx response containing at least one directive. How to fix it: Publish a robots.txt at your domain root declaring your sitemap and any crawler rules. ### robots.txt rules can be read https://llmstxt.studio/checks/robots-txt-invalid What: Checks that your robots.txt contains directives a crawler can actually act on. Why it matters: A robots.txt that parses to nothing is treated as absent. Worse, rules written before any User-agent line are silently ignored — so a file that looks restrictive can be permitting everything, or a file meant to welcome crawlers can be doing nothing at all. How we detect it: We parse the file per RFC 9309 and require at least one user-agent group. We also detect an HTML page served in place of a text file. How to fix it: Ensure every Allow/Disallow rule follows a User-agent line, and that robots.txt is served as text/plain. ## AI Discovery checks ### llms.txt is reachable https://llmstxt.studio/checks/llms-txt-missing What: Checks whether a file is served at /llms.txt on your domain root. Why it matters: llms.txt is a short, curated map of your site written for machines. Without one, an AI system has to crawl and infer your structure from raw pages — which is slower, lossier, and entirely at its discretion which pages it reads. How we detect it: We request https://yourdomain/llms.txt and require a 2xx response whose body is not an HTML page. Single-page apps often answer any unknown path with their own HTML shell; that is not a file, and we do not count it as one. How to fix it: Publish a plain-text Markdown file at /llms.txt with an H1 title, a one-line blockquote summary, and H2 sections listing your most important pages as links. ### llms.txt follows the spec https://llmstxt.studio/checks/llms-txt-invalid What: Checks that your llms.txt has the structure the llmstxt.org spec defines: an H1 title, a blockquote summary, and at least one section of links. Why it matters: Parsers read llms.txt positionally. A file missing its title or summary still downloads fine but yields nothing useful when parsed, so the effort of publishing it is wasted. How we detect it: We parse the file and require a level-one heading, a blockquote immediately following it, and at least one H2 section containing link items. How to fix it: Open with "# Your Site Name", follow it with "> a one-sentence description", then group links under "## Section" headings. ### llms.txt lists enough pages https://llmstxt.studio/checks/llms-txt-too-few-links What: Checks that your llms.txt links to at least three pages. Why it matters: The file exists to route a machine to your important content. One or two links routes it almost nowhere, and it will crawl the rest of your site anyway. How we detect it: We count link items across every section of the parsed file. How to fix it: Add your documentation, pricing, product and about pages — anything you would want quoted. ### llms.txt is small https://llmstxt.studio/checks/llms-txt-oversized What: Checks that your llms.txt is small enough to help rather than hurt — under 40KB, about 10,000 tokens. Why it matters: llms.txt is meant to be an index, not the content itself. In our read-cost benchmark a file under roughly 10,000 tokens made a site about twice as cheap for an AI agent to read; a file over it made the site about nine times more expensive than having no file at all. The detail belongs in llms-full.txt. How we detect it: We measure the UTF-8 byte length of the file. Over 40KB warns; under 200 bytes is treated as an empty stub. The threshold comes from the read-cost benchmark, not from the spec, which sets no limit. How to fix it: Keep llms.txt to titles, one-line descriptions and links. Move full page content into llms-full.txt. The AI Read Cost tool at /read-cost/ shows which lines to cut, with a reason for each, and the file it would leave. ### llms-full.txt is published https://llmstxt.studio/checks/llms-full-txt-missing What: Checks for an optional llms-full.txt containing your full content in one file. Why it matters: Where llms.txt is an index, llms-full.txt lets an AI system load your entire documented content in a single request rather than crawling page by page. How we detect it: We request https://yourdomain/llms-full.txt and require a 2xx response that is not an HTML page. How to fix it: Generate a single Markdown file containing the full text of the pages listed in your llms.txt, and serve it at /llms-full.txt. ### A sitemap is reachable https://llmstxt.studio/checks/sitemap-missing What: Checks that an XML sitemap is reachable, either declared in robots.txt or at the conventional /sitemap.xml. Why it matters: A sitemap is the complete list of what you want found. Without one, a crawler discovers pages only by following links, so anything weakly linked — new pages especially — may never be seen at all. How we detect it: We take the first Sitemap: URL declared in robots.txt, or fall back to /sitemap.xml, and require a 2xx response containing XML. How to fix it: Generate an XML sitemap, serve it at /sitemap.xml, and declare it in robots.txt with a Sitemap: line. ### The sitemap in robots.txt works https://llmstxt.studio/checks/sitemap-unreachable What: Checks that the Sitemap: URL declared in your robots.txt actually resolves. Why it matters: A declared sitemap that 404s is worse than none at all: a crawler follows the declaration, gets nothing, and has no reason to look for one anywhere else. How we detect it: We request the first Sitemap: URL from robots.txt and require a 2xx XML response. How to fix it: Correct the Sitemap: URL in robots.txt, or publish a sitemap at the address it names. ### robots.txt declares your sitemap https://llmstxt.studio/checks/sitemap-not-declared-in-robots What: Checks that robots.txt contains a Sitemap: line pointing at your sitemap. Why it matters: robots.txt is the first file a crawler reads, and the Sitemap: line is the standard way to hand it your full page list. Relying on the conventional /sitemap.xml path works only for crawlers that guess. How we detect it: We look for at least one Sitemap: directive in robots.txt. How to fix it: Add a "Sitemap: https://yourdomain.com/sitemap.xml" line to robots.txt. ### Homepage declares a canonical URL https://llmstxt.studio/checks/canonical-missing What: Checks for a on your homepage. Why it matters: Most sites are reachable at several addresses — with and without www, with tracking parameters, with and without a trailing slash. The canonical tag says which one is the real address, so a system citing you links to a URL that will still resolve in a year. How we detect it: We read from the parsed homepage and require a non-empty, absolute URL. How to fix it: Add to your document head, using the absolute preferred URL. ### llms.txt links work https://llmstxt.studio/checks/llms-txt-links-broken What: Samples the links in your llms.txt and checks that they resolve. Why it matters: A broken link in llms.txt is worse than a broken link on a page. This is the file you offered as your curated map, so a system following it lands on nothing — and has no reason to believe the rest of the file is any more accurate. How we detect it: We take up to ten links from your file, spread across its sections, and issue a HEAD request to each. Servers that refuse HEAD are not counted as broken. We deliberately sample rather than check everything, so an audit never becomes a crawl of your site. How to fix it: Correct or remove the dead links, and regenerate llms.txt whenever the pages it points at move. ### llms.txt covers your main pages https://llmstxt.studio/checks/key-pages-missing-from-llms-txt What: Compares the top-level pages in your sitemap against the links in your llms.txt. Why it matters: llms.txt is a curated subset, so it should not list everything. But if none of your shallowest, most important pages appear in it, the curation has drifted from the site — usually because the file was written once and the site kept moving. How we detect it: We take the shallowest URLs from your sitemap, ignoring deep or dated paths, and check how many appear among your llms.txt links. Fewer than a third warns. How to fix it: Add your primary pages — docs, pricing, product, about — to llms.txt, or regenerate it from the current sitemap. ## Content Understanding checks ### Homepage has a descriptive title https://llmstxt.studio/checks/title-tag-weak What: Checks that your homepage has a that identifies what the site is. Why it matters: The title is the single strongest signal of what a page is about. An AI system summarising or citing your site leans on it heavily, and a missing or generic title ("Home", "React App") tells it nothing it can use. How we detect it: We read the <title> element from the parsed homepage and require it to be present, at least 10 characters, and not one of the common placeholder values frameworks emit by default. How to fix it: Set a title naming the organisation and what it does — "Acme — inventory software for independent retailers" rather than "Home". ### Homepage has a meta description https://llmstxt.studio/checks/meta-description-missing What: Checks for a <meta name="description"> on your homepage. Why it matters: The description is a summary you control. Without one, any AI system describing your site writes its own from whatever body text it finds first — often navigation or a cookie banner. How we detect it: We read <meta name="description"> from the parsed homepage and require non-empty content. Attribute order does not matter. How to fix it: Add a one- or two-sentence description of what your organisation does. ### Homepage has exactly one H1 https://llmstxt.studio/checks/h1-missing-or-duplicated What: Checks that your homepage has exactly one <h1>. Why it matters: The H1 is the in-page statement of what this page is. None leaves a machine inferring it from layout; several leave it unable to tell which one is the subject. How we detect it: We count <h1> elements in the parsed homepage. How to fix it: Use a single <h1> for the page subject and <h2>/<h3> for everything beneath it. ### Heading levels are in order https://llmstxt.studio/checks/heading-hierarchy-broken What: Checks that heading levels on your homepage descend without skipping — an h2 does not jump straight to an h4. Why it matters: Headings are the outline a machine uses to work out which text belongs to which topic. A skipped level breaks the nesting, so content ends up attributed to the wrong section — or to no section at all. How we detect it: We read every heading on the homepage in document order and flag any jump of more than one level downward. How to fix it: Use the next level down rather than choosing a heading for its size, and style it with CSS instead. ### Page marks its main content https://llmstxt.studio/checks/semantic-html-missing What: Checks that your homepage uses semantic landmark elements — main, header, nav, footer or article — rather than only div and span. Why it matters: Landmarks tell a machine which part of the page is the content and which is furniture. Without them, navigation, cookie notices and footers are indistinguishable from what you actually wanted read, and often get quoted instead. How we detect it: We look for main, article, header, nav, footer and section elements on the homepage, and require at least two distinct kinds including a main or article. How to fix it: Wrap the primary content in <main>, and mark navigation, header and footer with their own elements. ### Page has enough readable text https://llmstxt.studio/checks/thin-content What: Checks that your homepage contains a meaningful amount of readable text, rather than carrying its message entirely in images or video. Why it matters: A machine reads text. A homepage whose proposition lives in a hero image and a video is, to an AI system, a page that says almost nothing — and it will describe you from whatever fragments it can find. How we detect it: We strip script, style, navigation and footer content from the homepage and count the remaining words. Under 100 warns. How to fix it: Put your core proposition in text — a headline and a paragraph or two of real prose — rather than only inside images. ### Article declares an author https://llmstxt.studio/checks/authorship-missing What: Checks that a page presenting itself as an article names its author in machine-readable form. Why it matters: Attribution is part of what makes a claim quotable. A system deciding whether to cite a piece of writing wants to know who stands behind it, and a name in the byline image or in prose alone is not something it can rely on. How we detect it: We only judge pages that declare themselves articles, via og:type or Article schema. On those we look for an author in JSON-LD, or a meta author tag. How to fix it: Add an "author" property to your Article schema, naming a Person or Organization. ### Article declares a publication date https://llmstxt.studio/checks/publish-date-missing What: Checks that a page presenting itself as an article carries a machine-readable publication date. Why it matters: AI systems weigh recency heavily when choosing what to cite. Undated content is treated as of unknown age, which in practice means it loses to anything that can prove it is current. How we detect it: We only judge pages that declare themselves articles. On those we look for datePublished in JSON-LD, an article:published_time meta tag, or a <time datetime> element. How to fix it: Add "datePublished" to your Article schema in ISO 8601 form, and "dateModified" when you revise it. ## Machine-Readable Data checks ### Homepage publishes JSON-LD structured data https://llmstxt.studio/checks/jsonld-missing What: Checks for at least one valid JSON-LD block on your homepage. Why it matters: Structured data states facts about you unambiguously — your name, what you sell, your prices — rather than leaving them to be inferred from prose. It is the difference between an AI system guessing at your details and reading them. How we detect it: We parse every <script type="application/ld+json"> on the homepage. Blocks wrapped in @graph are unwrapped. A block that fails to parse is reported by schema-invalid-syntax instead. How to fix it: Add a JSON-LD block declaring at minimum an Organization with your name, URL and logo. ### Structured data can be read https://llmstxt.studio/checks/schema-invalid-syntax What: Checks that every JSON-LD block on your homepage is valid JSON. Why it matters: A malformed block is discarded in full. The markup is there, it looks right in the page source, and it contributes nothing — which is worse than having none, because nobody goes looking for a problem they think they have already solved. How we detect it: We JSON.parse each <script type="application/ld+json"> and record any that throw. How to fix it: Validate the block — trailing commas and unescaped quotes inside strings are the usual causes. ### Organization schema is declared https://llmstxt.studio/checks/organization-schema-missing What: Checks for an Organization (or LocalBusiness) JSON-LD block on your homepage. Why it matters: This is how an AI system learns who you are as an entity rather than as a web page — your legal name, your logo, your social profiles. It is the anchor everything else about you attaches to. How we detect it: We look for a JSON-LD block whose @type is Organization, LocalBusiness, Corporation or one of their common subtypes. How to fix it: Add an Organization block with name, url, logo and sameAs links to your official profiles. ### Organization schema is complete https://llmstxt.studio/checks/schema-missing-recommended-fields What: Checks that your Organization schema carries the fields that make it useful — url, logo and sameAs. Why it matters: A bare Organization block says you exist. The url, logo and sameAs links are what connect that entity to your website, your brand mark and your verified profiles elsewhere — the evidence a system uses to be confident the organisation it is describing is actually you. How we detect it: We read the Organization or LocalBusiness block and check for url, logo and a non-empty sameAs. How to fix it: Add url, logo and sameAs (your official profiles) to the Organization block. ### Article pages declare Article schema https://llmstxt.studio/checks/article-schema-missing What: Checks that a page presenting itself as an article also declares Article, BlogPosting or NewsArticle schema. Why it matters: Article schema turns a page of prose into a citable object with an author, a date and a headline. Without it a system has the text but none of the provenance, and provenance is most of what decides whether something gets quoted. How we detect it: We only judge pages that declare themselves articles via og:type. On those we look for Article, BlogPosting or NewsArticle in the JSON-LD. How to fix it: Add an Article block with headline, author, datePublished and dateModified. ### Product schema gives a price https://llmstxt.studio/checks/product-schema-invalid What: Checks that any Product schema on the page includes an offers block carrying a price and a currency. Why it matters: A Product without a price is a product a system cannot compare, quote or recommend. It is also the most common structured-data mistake on commerce sites, because the markup validates perfectly while omitting the one field anyone wants. How we detect it: We find blocks with @type Product and require an offers property carrying price and priceCurrency. How to fix it: Add an offers object with price, priceCurrency and availability. ### FAQ content is marked up as FAQPage https://llmstxt.studio/checks/faq-schema-opportunity What: Checks whether a page that reads like a list of questions declares FAQPage schema. Why it matters: Question-and-answer content is the shape AI answers are built from. Marking it up hands a system ready-made pairs instead of asking it to infer which paragraph answers which heading. How we detect it: We look for three or more headings phrased as questions. If we find them and there is no FAQPage schema, we suggest it. Pages without question headings are skipped. How to fix it: Wrap the questions and answers in FAQPage schema with a mainEntity array of Question objects. ### Open Graph tags are present https://llmstxt.studio/checks/open-graph-missing What: Checks for og:title, og:description and og:image on your homepage. Why it matters: Open Graph is a second, independent statement of what a page is — widely consumed, trivially parsed, and useful to any system building a preview or a citation card for you. How we detect it: We read og:title, og:description and og:image from the parsed homepage and require all three. How to fix it: Add the three og: meta tags to your document head. ## Answer Readiness checks ### Your site gives an MCP address https://llmstxt.studio/checks/mcp-endpoint-missing What: Checks whether your site declares a Model Context Protocol endpoint that AI agents can call to take actions. Why it matters: Reading is only half of what agents now do. Without a declared endpoint, an agent can describe your product but cannot book, buy, search or submit anything on your behalf — so a task that could have completed on your site completes on a competitor's. How we detect it: We look for a manifest at /.well-known/mcp.json, and for a reference to MCP or a Model Context Protocol server inside your llms.txt. How to fix it: Expose an MCP server for the actions your site supports, publish a manifest at /.well-known/mcp.json, and reference it from your llms.txt. ### Canonical address is stable https://llmstxt.studio/checks/unstable-urls What: Checks that your canonical URL carries no session identifiers or campaign tracking parameters. Why it matters: A citation is only as durable as the URL inside it. A canonical carrying a session id points at something that expires, and one carrying campaign parameters attributes every future visitor to a campaign that ended — while quietly fragmenting the page across dozens of near-identical addresses. How we detect it: We read the canonical URL and look for session or tracking parameters: utm_*, gclid, fbclid, sessionid, PHPSESSID, jsessionid, sid, and similar. How to fix it: Strip query parameters from the canonical URL. It should be the plain address of the page. ### Articles link to their sources https://llmstxt.studio/checks/source-attribution-missing What: Checks whether a page presenting itself as an article links out to any source at all. Why it matters: Content that cites nothing is harder to trust and harder to corroborate. A system weighing whether to repeat a claim looks for whether the claim is anchored to anything outside the page making it. How we detect it: We only judge pages declaring themselves articles via og:type. On those we count links to domains other than your own, ignoring social share links and navigation. How to fix it: Link to the primary sources behind your factual claims — the study, the documentation, the announcement. ### Your text is in the HTML https://llmstxt.studio/checks/content-client-rendered What: Compares the content in your HTML as delivered against what a browser produces after running your JavaScript. Why it matters: This is the most consequential failure on the list, and the least visible. The page looks perfect in your browser because your browser runs the JavaScript. Many AI crawlers do not — they read the HTML as delivered. If your content arrives only after a bundle executes, those systems see an empty container. How we detect it: We first look at the HTML exactly as served: how much readable text it contains, whether it has an empty framework mount, and how many scripts it loads. When that is unambiguous we stop there. When it is not, we render the page in a headless browser and compare the text a browser ends up with against the text that was delivered. Rendering is time-boxed; if it cannot complete we fall back to the delivered-HTML signature and say so. How to fix it: Server-render or pre-render your pages so the content is in the HTML. Every major framework supports this — Next.js, Nuxt, SvelteKit and Remix all render on the server by default.