By Iliass El Barhoumi · Published · Last updated
What is Claude-SearchBot?
Claude-SearchBot is the crawler Anthropic uses to improve the quality of Claude’s search results, analysing pages so its answers are relevant and accurate. It is Anthropic’s equivalent of OAI-SearchBot and PerplexityBot: the token that decides whether Claude can find and cite your page when it answers with web results.
| Operator | Anthropic |
|---|---|
| Purpose | search index · Claude’s search answers |
| robots.txt token | Claude-SearchBot |
| In the user-agent | Claude-SearchBot |
| CrawlRadar weight | 2 of the crawler-access weights, and a block is flagged critical |
| Vendor docs | Anthropic: Does Anthropic crawl data from the web? |
How to allow Claude-SearchBot in robots.txt
Give Claude-SearchBot its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: Claude-SearchBot
Allow: /How to block Claude-SearchBot in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: Claude-SearchBot
Disallow: /- Tokens are matched case-insensitively, but writing
Claude-SearchBotexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block Claude-SearchBot without any line in robots.txt.
Should you block Claude-SearchBot?
Allow Claude-SearchBot if you want Claude to cite you. Anthropic’s documentation says disabling it prevents your content from being indexed for search and may reduce your visibility and accuracy in Claude’s search results. Training is a separate decision you make with ClaudeBot.
How to verify a request is really Claude-SearchBot
Anthropic publishes the IP addresses Claude-SearchBot uses at https://claude.com/crawling/bots.json. A request that claims to be Claude-SearchBot from any other address is someone borrowing the name, and should be treated like any other scraper. CrawlRadar’s collector checks every hit against that list and labels it verified or spoofed.
Anthropic’s other AI crawlers
Each Anthropic token is a separate robots.txt decision. Blocking one has no effect on the others.
| Token | Purpose | What blocking it costs |
|---|---|---|
ClaudeBot | training | Blocking it keeps your future pages out of Anthropic’s training data. Claude’s search citations come from Claude-SearchBot, which is unaffected. |
Claude-User | live answer fetch | It fetches your page when someone asks Claude a question that needs it. Blocking it means Claude cannot retrieve your page for that person. |
anthropic-ai | training | A legacy Anthropic token. Worth keeping in your robots.txt alongside ClaudeBot so an older group cannot contradict a newer one. |
Is Claude-SearchBot visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each Claude-SearchBot visit, the page it fetched, and whether it came from Anthropic. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
Is Claude-SearchBot the same as ClaudeBot?
No. ClaudeBot collects content that may be used for training; Claude-SearchBot indexes pages for Claude’s search answers. You can allow one and block the other, and each obeys only its own robots.txt group.
How do I allow Claude-SearchBot but block ClaudeBot?
Give each token its own group: “User-agent: Claude-SearchBot” followed by “Allow: /”, and “User-agent: ClaudeBot” followed by “Disallow: /”. A named group overrides the wildcard group, so the order of the two groups in the file does not matter.