AI Read Cost — Methodology v1.1.0

The question this benchmark answers, stated as narrowly as we can make it:

How much retrieval work does an agent-like system need to perform to find enough authoritative information to answer a question about a website?

It does not measure whether llms.txt improves AI rankings, whether any named assistant reads llms.txt, or what AI companies spend. Those claims are outside what this experiment can support. See "What we never claim" below.

The benchmark unit

One task is a domain + question + retrieval route. Every number we publish is an aggregate over tasks of that shape.

Questions (v1)

Five questions, each with a rubric used for human calibration and by the automated judge. Ids are fixed and never renumbered.

They were chosen because every commercial website is expected to answer all five, and because the answers sit at different depths. A question answerable from the homepage alone cannot separate a good llms.txt from a bad one.

Full text and rubrics: questions.ts.

Retrieval routes

Each route answers the same question against the same live site, differing only in how it discovers what to fetch.

• html — start at /, follow in-page links, same host only.
• sitemap — read robots.txt and /sitemap.xml, select likely-relevant URLs.
• llms — read /llms.txt, follow the resources it lists.
• llms-full — read /llms-full.txt; the file itself is the evidence.
• generated — a candidate llms.txt built by us from the sitemap and never deployed, run through the llms logic.

llms-full is kept separate from llms deliberately. Merging them would let one large file flatter the llms.txt result and hide which representation did the work.

generated is a comparison involving our own product. Any output including it says so. Observational results and product comparisons are never mixed into a single aggregate.

Stopping conditions

A route stops when one of these is true, and the reason is recorded:

20 requests sits above what a well-structured site needs for any of the five questions, and below the point where a badly structured site would run indefinitely. A route that hits the ceiling has told us something real, and the stop reason records it rather than hiding it.

Constants: config.ts.

Measurements

Every route records the same set.

Tokens to Answer is null, not equal to Tokens Retrieved, when a route never reached sufficiency. Reporting it otherwise would make an unsuccessful route look merely expensive.

Failed and redirected requests count. They are retrieval work.

Useful Token Ratio is measured per request, not per quoted snippet

A snippet-level reading looks defensible and is not. A 40,000-token retrieval whose answer is one sentence would score about 0.05%, and so would a 400-token retrieval whose answer is the same sentence. The metric would rank every route as equally wasteful and discriminate nothing.

The question it should answer is how much of what was pulled down needed to be. So it is the token count of the requests that produced evidence over the token count of every request made. Fetch five pages, find the answer in one, and that is 20%.

The snippet-level figure remains recoverable from the stored evidence when information density is what is being asked; it is simply not this metric.

Tokenisation

One fixed encoding, cl100k_base, for all comparisons. Dollar figures, where shown at all, are derived from this count for display only. They are never a second measurement and never the headline.

Answer quality

Efficiency without accuracy is meaningless: retrieving nothing is maximally cheap. Every route therefore produces an answer, generated only from the evidence that route retrieved, with outside knowledge forbidden.

Answers, source URLs and supporting passages are stored so any published number traces back to the bytes that produced it.

The answer is authoritative, not the sufficiency check

Retrieval stops when a sufficiency judge says the evidence answers the question. That judge is not trusted to set the headline metric on its own. In piloting it accepted a shell snippet containing the word COST as evidence of a price, and a blog excerpt as a description of an organisation — roughly one attempt in ten, each one stopping retrieval and recording a Tokens to Answer for a question that was never answered.

So the engine proves the verdict: when the judge says "answerable", it immediately produces the answer from the passages cited. If it cannot, the verdict is discarded and retrieval continues, as an agent's would.

Tokens to Answer can therefore only be set by an attempt that produced an answer. Discarded verdicts are counted per route as falseSufficiency — a health metric for the benchmark, not a property of the website. A rising count means the judge has drifted and the prompt needs work.

Judge calibration

The automated judge is not trusted until measured against human labels. The first 100 (domain, question, route) results are labelled by hand, agreement with the judge is computed, and the figure is recorded here before any aggregate is published.

Agreement with human labels: not yet measured. No aggregate answer-quality claim may be published until this line carries a number.

Comparison rule

Any output showing a route's efficiency shows its answer quality in the same block. This is enforced by the type system: a comparison row without a quality grade does not compile.

Our claims therefore take the form:

llms.txt reduced median Tokens to Answer by 64% while maintaining comparable answer accuracy.

and never:

llms.txt uses 64% fewer tokens.

The generated route

generated benchmarks the llms.txt that llmstxt.studio would produce for a site. The file is built in memory from the site's sitemap — the same selection and assembly as the product's preview (fetchSitemap → rankUrls → assembleMarkdown, 15 links, generic or Shopify layout) — and then run through exactly the llms route logic. Link fetches go to the live site. The file is never deployed and the site is never told. For a site that already has an llms.txt, both llms (theirs) and generated (ours) run and both are stored; the label is always "generated (not deployed)".

This is a comparison involving our own product, not an observation of the web. It is excluded from OBSERVATIONAL_ROUTES, it is not in the default route set and must be requested explicitly, and every output that includes it carries the disclosure: "Includes a comparison involving a file generated by llmstxt.studio, which is not deployed on this site." Observational results and this route are never combined into one aggregate.

Ledger accounting

Generation cost is not retrieval cost. The sitemap requests used to build the candidate are not charged to the route. The reasoning: the route models a site that has deployed this file, and an agent reading a deployed file would not perform them. Reading the file is charged: one synthetic ledger entry,

url            generated://<domain>/llms.txt
status         200
contentType    text/plain
rawBytes       UTF-8 byte length of the generated text
rawTokens      tokens of the generated text (fixed encoding)
textTokens     tokens of the generated text after whitespace collapse
elapsedMs      0 (no request was made; a made-up latency would be worse)
discoveredVia  generated

Every page fetched by following a link from the file is a real request to the live site, charged in full and labelled discoveredVia: generated.

A reader may reasonably disagree with this rule. Two alternatives: charge the sitemap fetches as well (the route would then look strictly worse than sitemap, since it performs the same discovery and then reads a file too), or charge nothing for the synthetic read (the route would then be flattered by a free index). We chose the middle, stated it here, and the ledger keeps enough to recompute either alternative: the synthetic entry is identifiable by its generated:// scheme and the sitemap route's ledger for the same domain records what discovery cost.

generatedTokens

Phase 1 found the size of the llms.txt is the decisive variable: files under roughly 10k tokens beat conventional retrieval, files over it lost. The generated route is therefore only interesting when the generator produces something small, and its size must be visible in every result. It is recorded as generatedTokens on the candidate and is recoverable from any stored run as textTokens on the ledger entry whose url begins generated://. The rawBytes of that entry is the file size in bytes.

If the sitemap cannot be fetched, or lists zero URLs, the route is route-absent — the product could not have produced a file for that site either. It is not an error and not a failure of the site.

Snapshots

Every run stores content hashes of the homepage, sitemap, llms.txt and llms-full.txt — and of the generated candidate, when that route ran — so a later change in the numbers can be attributed to the website changing rather than to us changing the benchmark.

Versioning

BENCHMARK_VERSION is stored on every run. It bumps when any of these change: the question set, the routes or their selection logic, the stopping conditions, or the tokenizer/extractor. Historical runs keep the version they ran under and are never migrated.

Current: 1.1.0. Public tool: 1.1.0-public (2 questions, 2 routes, tighter limits — not comparable with research runs).

1.1.0 added a per-question sufficiency criterion. Under 1.0.0 one rule served all five questions and rejected mechanisms — a contact form, a sign-up button, a product category list — because they are not quotable sentences. Phase 1 showed 5 of 5 attempts that fetched a get-started page and 17 of 26 that fetched a contact page still returned "unable". Each question now states what counts as sufficient for it. Phase 1 (1.0.0) and phase 2 (1.1.0) are therefore not numerically comparable on answer rates; the between-route efficiency ratio, being a within-question comparison under one judge, is expected to carry over.

What we never claim

• That llms.txt improves AI rankings or visibility.
• That ChatGPT, Claude, Gemini or any other assistant reads llms.txt.
• That our token counts equal the internal cost incurred by any AI company.
• That fewer tokens is automatically better, absent answer quality.
• A single unexplained 0–100 "AI readiness" score derived from these metrics.

Limitations

• We do not emulate any proprietary crawler architecture. This is an agent-like retrieval system of our own construction, and results describe it.
• Relevance selection is a heuristic. A different heuristic would produce different absolute numbers; routes are compared under the same one.
• One tokenizer stands in for many. Ratios between routes are more meaningful than absolute token counts.
• Sites change. Snapshot hashes let us detect this but not undo it.
• Discovery is charged to the route that performs it. Reading three sitemap index files before reaching a URL list is work an agent genuinely does, and a site that spreads its URLs across many files reads as more expensive. This is intended, and it is a choice a reader may disagree with.
• The sitemap route reads at most five sitemaps declared in robots.txt and follows an index one level. A site declaring more than that is measured on what a bounded reader would see, not on its full declaration.