All checks / AI Access
Error
bot-challenge-blocking-agentsNo bot challenge blocks AI systems
What this check means
Checks whether your homepage answers an identified bot with a challenge or a refusal rather than your content.
Why it matters
This is the quiet one. Your robots.txt can welcome every AI crawler while your CDN turns them away at the door, and nothing in your configuration will tell you — the rules that permit them and the rules that block them live in different systems.
How we detect it
We request your homepage with an identifiable bot user-agent and look for 401, 403 or 429, or a Cloudflare-style interstitial (a cf-mitigated header, or a 503 from a Cloudflare edge).
How to fix it
Allow-list the AI crawlers you intend to permit in your CDN or WAF bot-management rules, not only in robots.txt.
Check whether this applies to your site.
Run an AI audit