llmstxt.studio-registry-bot

You saw our bot in your logs

It asked your site one question: do you publish an llms.txt file? That is the whole visit. Here is exactly what it does, so you can decide whether you mind.

What it requests

  • One URL only: https://yourdomain/llms.txt. It never crawls pages, follows links, or reads anything else.
  • A HEAD request first. If that says the file is not there, it stops. It sends a GET only when a file appears to exist.
  • Identifies itself as llmstxt.studio-registry-bot/1.0 (+https://llmstxt.studio/bot).
  • Only ever over https. If your site redirects it to plain http, it stops there and records the redirect instead of following it.
  • Gives up after six seconds. A site that did not answer in time is asked once more later in the same sweep, then left until the next one.

How often

  • Sites that publish a file: about once a week, to record when it changes.
  • Sites that do not: about once a month.
  • At most one request in flight to any domain at a time.

What it keeps

Whether the file exists, its size, how many links it lists, when it last changed, and a copy of the file itself when there is one. This builds the public llms.txt registry and the adoption statistics. Nothing about your visitors, your pages or your users is requested or stored.

How to opt out

  • Return 403 or 404 for /llms.txt. The bot records that and backs off to the monthly check.
  • Block the user agent llmstxt.studio-registry-bot at your edge or in your server configuration.
  • Or email [email protected] with your domain and we will remove it from the list by hand.

Why we do this: llms.txt is a proposed convention with no public record of who has adopted it or how the files are changing. The registry is that record, built from what domains actually serve, published openly, and counted honestly. Our methodology, and what the record says so far about whether the file is used: Does llms.txt work?