AI crawler directory
The AI crawlers your robots.txt decides for
These are the 12 crawler tokens CrawlRadar reads your robots.txt for, one group at a time, for the exact path scanned — 5 of them belong to a primary answer engine, and blocking one of those removes you from that engine’s citation pool entirely.
Check my robots.txt- 12 tokens checked
- 5 primary engine bots
- CrawlRadar checks GPTBot. OpenAI (training + search index)
GPTBot - CrawlRadar checks OAI-SearchBot. ChatGPT SearchOAI-SearchBot
- CrawlRadar checks ChatGPT-User. ChatGPT live browsingChatGPT-User
- CrawlRadar checks ClaudeBot. Anthropic / Claude
ClaudeBot - CrawlRadar checks anthropic-ai. Anthropic (legacy UA)anthropic-ai
- CrawlRadar checks PerplexityBot. Perplexity index
PerplexityBot - CrawlRadar checks Perplexity-User. Perplexity live fetchPerplexity-User
- CrawlRadar checks Google-Extended. Google AI Overviews / Gemini
Google-Extended - CCBotCrawlRadar checks CCBot. Common Crawl (feeds many models)
- CrawlRadar checks Bytespider. ByteDance
Bytespider - CrawlRadar checks Amazonbot. Amazon / Alexa
Amazonbot - Applebot-ExtendedCrawlRadar checks Applebot-Extended. Apple Intelligence