Is my site blocking GPTBot?

Enter your URL above to find out. The most common cause is not a deliberate block but a plugin or a CDN default: Cloudflare’s "Block AI Scrapers and Crawlers" toggle, Wordfence, and several WordPress SEO plugins add AI crawler Disallow rules without saying so in a way most site owners notice.

What is the difference between GPTBot and OAI-SearchBot?

OpenAI documents GPTBot as its model-training crawler and OAI-SearchBot as the crawler that surfaces sites in ChatGPT Search. The two settings are independent: blocking GPTBot while allowing OAI-SearchBot means no training but yes to citation, and CrawlRadar reports a GPTBot block as an opt-out rather than a blocking issue for exactly that reason.

Does blocking Google-Extended remove me from Google search?

No, and it does not remove you from AI Overviews either. Google-Extended controls whether content Google crawls may be used to train Gemini models and to ground Gemini Apps and Vertex AI answers. Google states it does not affect inclusion in Google Search, and AI Overviews are a Search feature built from Googlebot’s crawl.

Should I allow every AI crawler?

That depends on what you sell. If your business benefits from being cited as a source, whether SaaS, documentation, services or media with a subscription funnel, then blocking the crawlers that feed answer engines removes you from an entire discovery channel. If your content is the product and it is sold per-view, blocking training crawlers while allowing search crawlers is a defensible middle position.

Do AI crawlers actually obey robots.txt?

The major ones publish their user-agent tokens and honour robots.txt: OpenAI, Anthropic, Perplexity and Google all document theirs. Compliance is voluntary and unenforceable, and there have been credible reports of less scrupulous crawlers ignoring it. robots.txt is the mechanism for the crawlers you want to influence, not a security control.

How long until a robots.txt change takes effect?

Crawlers cache robots.txt, typically for up to 24 hours. Expect a day or so before a change is reflected, and longer before content crawled under the old rules works its way through an index.

Free tool · no account

AI crawler robots.txt checker

See which of 14 AI crawlers your robots.txt allows or blocks, and the exact lines to change.

Free, no sign-up. Reads only the site's /robots.txt.

How it works

1Enter your domain
2We read your robots.txt like each crawler does
3See who is blocked, and the fix

What it checks

14 AI crawlersGPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest.
Search vs trainingOnly 3 decide citations. The rest are a safe opt-out.
The exact lines to changeCopy-ready robots.txt rules for every blocked crawler.

Frequently asked questions

Is my site blocking GPTBot?
Enter your URL above to find out. The most common cause is not a deliberate block but a plugin or a CDN default: Cloudflare’s "Block AI Scrapers and Crawlers" toggle, Wordfence, and several WordPress SEO plugins add AI crawler Disallow rules without saying so in a way most site owners notice.
What is the difference between GPTBot and OAI-SearchBot?
OpenAI documents GPTBot as its model-training crawler and OAI-SearchBot as the crawler that surfaces sites in ChatGPT Search. The two settings are independent: blocking GPTBot while allowing OAI-SearchBot means no training but yes to citation, and CrawlRadar reports a GPTBot block as an opt-out rather than a blocking issue for exactly that reason.
Does blocking Google-Extended remove me from Google search?
No, and it does not remove you from AI Overviews either. Google-Extended controls whether content Google crawls may be used to train Gemini models and to ground Gemini Apps and Vertex AI answers. Google states it does not affect inclusion in Google Search, and AI Overviews are a Search feature built from Googlebot’s crawl.
Should I allow every AI crawler?
That depends on what you sell. If your business benefits from being cited as a source, whether SaaS, documentation, services or media with a subscription funnel, then blocking the crawlers that feed answer engines removes you from an entire discovery channel. If your content is the product and it is sold per-view, blocking training crawlers while allowing search crawlers is a defensible middle position.
Do AI crawlers actually obey robots.txt?
The major ones publish their user-agent tokens and honour robots.txt: OpenAI, Anthropic, Perplexity and Google all document theirs. Compliance is voluntary and unenforceable, and there have been credible reports of less scrupulous crawlers ignoring it. robots.txt is the mechanism for the crawlers you want to influence, not a security control.
How long until a robots.txt change takes effect?
Crawlers cache robots.txt, typically for up to 24 hours. Expect a day or so before a change is reflected, and longer before content crawled under the old rules works its way through an index.

Read the full guideHow it works in detail, and its limits · 3 min read

The AI crawlers that matter, and what each one does

14 user-agents are worth tracking, and only 3 of them (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether an answer engine can cite you. CrawlRadar weights those twice as heavily as the rest and flags a block as critical. The training crawlers are an opt-out: blocking them keeps your pages out of model training without removing you from any engine’s answers.

AI crawler user-agents, their operators and the cost of blocking each
User-agent	Operator	Blocking it means
`GPTBot`	OpenAI	Blocking it keeps your pages out of future OpenAI model training. It does not remove you from ChatGPT Search, which crawls under OAI-SearchBot.
`OAI-SearchBot`	OpenAI	This is the crawler that decides whether ChatGPT Search can cite you. Blocking it removes you from that citation pool entirely.
`ChatGPT-User`	OpenAI	It fetches your page while someone is waiting on an answer. Blocking it means ChatGPT may not open your page even when a user asks it to.
`ClaudeBot`	Anthropic	Blocking it keeps your future pages out of Anthropic’s training data. Claude’s search citations come from Claude-SearchBot, which is unaffected.
`Claude-SearchBot`	Anthropic	This is the crawler that indexes pages for Claude’s search answers. Blocking it can remove you from the results Claude cites.
`Claude-User`	Anthropic	It fetches your page when someone asks Claude a question that needs it. Blocking it means Claude cannot retrieve your page for that person.
`anthropic-ai`	Anthropic	A legacy Anthropic token. Worth keeping in your robots.txt alongside ClaudeBot so an older group cannot contradict a newer one.
`PerplexityBot`	Perplexity	This is the crawler that builds Perplexity’s index. Blocking it takes you out of the pool its answers are drawn from.
`Perplexity-User`	Perplexity	It fetches your page in response to a live question. Blocking it costs you the highest-intent visit a crawler can make.
`Google-Extended`	Google	It is a robots.txt directive, not a crawler: no request ever arrives under this name. Disallowing it opts you out of Gemini training and grounding. It does not affect Google Search, including AI Overviews.
`CCBot`	Common Crawl	Common Crawl’s archive feeds many models indirectly. Blocking it is a broad training opt-out with a long tail.
`Bytespider`	ByteDance	A training crawler with a reputation for crawling aggressively. Blocking it is a common choice and costs little in Western answer engines.
`Amazonbot`	Amazon	Amazon uses it to improve its products, including Alexa’s answers, and says the content may be used to train its AI models.
`Applebot-Extended`	Apple	Like Google-Extended, this is a directive rather than a crawler: it opts you out of Apple’s model training while leaving plain Applebot, which powers Siri, Spotlight and Safari results, untouched.

How to allow AI crawlers in robots.txt

Add an explicit Allow: / group for each bot, above any broad Disallow that would otherwise catch it. Most robots.txt parsers apply the most specific matching group, so a named group beats a wildcard.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

That file allows the crawlers that decide citations and live answers. Add the training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) with Allow or Disallow depending on whether you want your pages used for model training. Either choice is valid.

Three mistakes account for most accidental blocks:

A blanket User-agent: * / Disallow: / left over from a staging environment.
A CDN or plugin toggle. Cloudflare’s AI-crawler block, in particular, operates above your robots.txt and will not appear in the file at all. This tool reads what the server serves, so it catches the file-level cases but not a WAF-level block.
A Disallow on the path, not the site. Blocking /blog/ for a wildcard group blocks it for every AI crawler too. This tool checks the exact path you enter, not just the root.

Blocking is not the only way to disappear

A page can be perfectly crawlable and still never be cited. robots.txt is the first gate, worth 8 of 100 points in the CrawlRadar model. The other 92 are about whether the page is parseable, attributable and quotable once a crawler has it.

Run the GEO score checker for the full picture, including whether your content exists before JavaScript runs and whether any section actually answers its own heading.

By Iliass El Barhoumi · Published 4 August 2026 · Last updated 1 October 2026