By Iliass El Barhoumi · Published · Last updated
What is ClaudeBot?
ClaudeBot is the crawler Anthropic uses to collect public web content that could contribute to training its models. Anthropic documents it alongside Claude-SearchBot, which indexes pages for search, and Claude-User, which fetches a page when a user asks Claude a question. Restricting ClaudeBot signals that your future content should be excluded from training.
| Operator | Anthropic |
|---|---|
| Purpose | training · Anthropic model training |
| robots.txt token | ClaudeBot |
| In the user-agent | compatible; ClaudeBot/1.0; +claudebot@anthropic.com |
| CrawlRadar weight | 1 of the crawler-access weights, never flagged critical |
| Vendor docs | Anthropic: Does Anthropic crawl data from the web? |
How to allow ClaudeBot in robots.txt
Give ClaudeBot its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: ClaudeBot
Allow: /How to block ClaudeBot in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: ClaudeBot
Disallow: /- Tokens are matched case-insensitively, but writing
ClaudeBotexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block ClaudeBot without any line in robots.txt.
Should you block ClaudeBot?
Block ClaudeBot to keep your pages out of Anthropic’s training data; it does not remove you from Claude’s search answers, which are governed by Claude-SearchBot. If the concern is server load rather than training, Anthropic says its bots support the non-standard Crawl-delay directive, so you can slow ClaudeBot down instead of blocking it.
How to verify a request is really ClaudeBot
Anthropic publishes the IP addresses ClaudeBot uses at https://claude.com/crawling/bots.json. A request that claims to be ClaudeBot from any other address is someone borrowing the name, and should be treated like any other scraper. CrawlRadar’s collector checks every hit against that list and labels it verified or spoofed.
Anthropic’s other AI crawlers
Each Anthropic token is a separate robots.txt decision. Blocking one has no effect on the others.
| Token | Purpose | What blocking it costs |
|---|---|---|
Claude-SearchBot | search index | This is the crawler that indexes pages for Claude’s search answers. Blocking it can remove you from the results Claude cites. |
Claude-User | live answer fetch | It fetches your page when someone asks Claude a question that needs it. Blocking it means Claude cannot retrieve your page for that person. |
anthropic-ai | training | A legacy Anthropic token. Worth keeping in your robots.txt alongside ClaudeBot so an older group cannot contradict a newer one. |
Is ClaudeBot visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each ClaudeBot visit, the page it fetched, and whether it came from Anthropic. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
Does blocking ClaudeBot stop Claude from citing my site?
No. Claude’s search answers draw on pages indexed by Claude-SearchBot, and live fetches come from Claude-User. ClaudeBot only governs whether your content may be used for training.
What is the difference between ClaudeBot and anthropic-ai?
anthropic-ai is an older token. ClaudeBot, Claude-SearchBot and Claude-User are the three Anthropic documents today. Listing anthropic-ai as well is harmless as long as its rule matches your ClaudeBot rule.