Retrieved · 8 of 100 points
Is your robots.txt blocking AI crawlers?
This check reads your robots.txt the way each agent does — group by group, for the exact path scanned — and reports which of the AI crawlers are allowed, because an engine that cannot fetch the page cannot cite it no matter how good the writing is.
- 8 points of 100
- 8/25 of the Retrieved lever
- crawler-access check id
Body copy pending
Phase 2 adds what ranks: how the check is scored, what a pass and a fail look like side by side, the evidence the report returns for it, and the fix in the order worth doing it. The lever this belongs to asks “Can an engine fetch this page at all?”
This page is noindex until that lands, and it is not in the sitemap.