AI crawler directory

# AI crawler list: all 14 AI bots and what each one does

These are the 14 AI crawler tokens CrawlRadar reads your robots.txt for, one group at a time, for the exact path scanned — and only 3 of them (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the search indexes answer engines cite from. The rest train models or fetch pages on request.

[Check my robots.txt](/tools/ai-crawler-checker)
- 14 tokens checked
- 3 search crawlers

By [Iliass El Barhoumi](/about) · Published 1 October 2026 · Last updated 1 October 2026

AI crawlers fall into **four groups**, and only one of them decides citations. **Search crawlers** build the indexes answer engines quote from; **live fetchers** open a page because someone asked; training crawlers and policy tokens decide whether your pages feed future models.

## Search crawlers: they decide citations

Blocking one removes you from that engine’s answers. CrawlRadar flags a block as critical.

- [![](/vendors/openai.svg)OAI-SearchBotOpenAI: ChatGPT Search index](/crawlers/oai-searchbot)
- [![](/vendors/anthropic.svg)Claude-SearchBotAnthropic: Claude search index](/crawlers/claude-searchbot)
- [![](/vendors/perplexity.svg)PerplexityBotPerplexity: Perplexity search index](/crawlers/perplexitybot)

## Live fetchers: a person asked for your page

Each request is a conversation that needed your page. Blocking them costs the highest-intent visits.

- [![](/vendors/openai.svg)ChatGPT-UserOpenAI: ChatGPT live page fetch](/crawlers/chatgpt-user)
- [![](/vendors/anthropic.svg)Claude-UserAnthropic: Claude live page fetch](/crawlers/claude-user)
- [![](/vendors/perplexity.svg)Perplexity-UserPerplexity: Perplexity live page fetch](/crawlers/perplexity-user)

## Training crawlers: an opt-out, not a blocker

Blocking them keeps your pages out of model training without removing you from any answer engine.

- [![](/vendors/openai.svg)GPTBotOpenAI: OpenAI model training](/crawlers/gptbot)
- [![](/vendors/anthropic.svg)ClaudeBotAnthropic: Anthropic model training](/crawlers/claudebot)
- [![](/vendors/anthropic.svg)anthropic-aiAnthropic: Anthropic (legacy token)](/crawlers/anthropic-ai)
- [CCBotCommon Crawl: Common Crawl (feeds many models)](/crawlers/ccbot)
- [![](/vendors/bytedance.svg)BytespiderByteDance: ByteDance model training](/crawlers/bytespider)
- [![](/vendors/amazon.svg)AmazonbotAmazon: Amazon (Alexa answers, may train models)](/crawlers/amazonbot)

## Policy tokens: read from robots.txt, never sent

No request carries these names. They control how the vendor’s regular crawl may be used.

- [![](/vendors/google.svg)Google-ExtendedGoogle: Gemini training and grounding (not AI Overviews)](/crawlers/google-extended)
- [Applebot-ExtendedApple: Apple Intelligence training](/crawlers/applebot-extended)

FAQ

## Frequently asked questions

What each kind of AI crawler does, and which ones decide whether you get cited.

- ### Which AI crawlers should I allow in robots.txt?  
Allow the search crawlers, OAI-SearchBot, Claude-SearchBot and PerplexityBot, if you want ChatGPT, Claude and Perplexity to cite you. Allowing or blocking the training crawlers, such as GPTBot, ClaudeBot and CCBot, is a separate choice about model training that does not affect citations.
- ### Is it safe to block GPTBot?  
Yes, if you do not want your pages used to train OpenAI’s models. OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the one behind ChatGPT Search, and states the two settings are independent.
- ### Does Google-Extended control Google AI Overviews?  
No. Google-Extended governs Gemini training and grounding, and Google states it does not affect inclusion in Google Search. AI Overviews are a Search feature built from Googlebot’s crawl.

---

Source: https://www.crawlradar.xyz/crawlers
