All checks / AI Access
Warning
robots-txt-invalidrobots.txt is parseable
What this check means
Checks that your robots.txt contains directives a crawler can actually act on.
Why it matters
A robots.txt that parses to nothing is treated as absent. Worse, rules written before any User-agent line are silently ignored — so a file that looks restrictive can be permitting everything, or a file meant to welcome crawlers can be doing nothing at all.
How we detect it
We parse the file per RFC 9309 and require at least one user-agent group. We also detect an HTML page served in place of a text file.
How to fix it
Ensure every Allow/Disallow rule follows a User-agent line, and that robots.txt is served as text/plain.
Check whether this applies to your site.
Run an AI audit