CrawlRadar

Retrieved · 8 of 100 points

Is your robots.txt blocking AI crawlers?

This check reads your robots.txt the way each agent does — group by group, for the exact path scanned — and reports which of the 14 AI crawler tokens are allowed, because an engine that cannot fetch the page cannot cite it no matter how good the writing is.

Score my page on thisSee all 14 checks and their weights →
  • 8 points of 100
  • 8/25 of the Retrieved lever
  • crawler-access check id

By · Published · Last updated

How is ai crawler access scored?

AI crawler access is worth 8 of 100 points, 8 of the 25 in the Retrieved lever, which asks “Can an engine fetch this page at all?” The engine scores it with these rules:

  • Each of the 14 tokens is checked against your robots.txt for the exact path scanned, using the most specific matching group.
  • Search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) carry double weight. Blocking one forces the check to warn and flags it as critical; blocking two or more fails it.
  • Training-only tokens (GPTBot, ClaudeBot, CCBot, Google-Extended and the like) carry single weight and are reported as an opt-out, never as a blocker, because blocking them does not stop any engine citing you.
  • No robots.txt means everything is allowed. A robots.txt that could not be fetched is reported as not checked, and its points leave the total rather than being awarded.

What does a pass and a fail look like?

VerdictExample
PassA robots.txt that allows OAI-SearchBot, Claude-SearchBot and PerplexityBot on every public path, and disallows GPTBot and CCBot because the owner opted out of training. Full search access, a deliberate opt-out.
FailA security plugin added “User-agent: OAI-SearchBot / Disallow: /” along with a list of scrapers. ChatGPT Search can no longer cite any page on the site, and nobody noticed.

How do you fix ai crawler access?

In the order worth doing it:

  1. Run the free AI crawler checker on the exact URL you care about, not just the home page.
  2. For each blocked search crawler, delete its Disallow group or replace it with “Allow: /”.
  3. Decide on training crawlers deliberately: blocking them is a valid choice, not an error.
  4. Check your CDN and security plugins, which can block crawlers at the firewall without touching robots.txt.

A scan reports the evidence behind the verdict, such as the sections, paragraphs or tokens that failed, and the fix prompt hands the same evidence to a coding agent so it can make the change for you.

Related checks in the Retrieved lever

Frequently asked questions

Which AI crawlers should I never block?

The search-index crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot. They build the indexes ChatGPT Search, Claude and Perplexity cite from, so blocking one removes you from that engine’s answers.

Is it bad to block GPTBot?

No. GPTBot is OpenAI’s training crawler, and blocking it is an opt-out from model training. ChatGPT Search uses OAI-SearchBot, so as long as that one is allowed you can still be cited.

Fix the page, then watch AI arrive

A better score makes a page eligible to be quoted. CrawlRadar then shows whether it happens: the AI crawlers that fetch the page, and the visitors ChatGPT, Perplexity and Claude send to it.

14-day free trial. No credit card.

Or start with the free AI crawler checker.