By Iliass El Barhoumi · Published · Last updated
What is anthropic-ai?
anthropic-ai is an older robots.txt token associated with Anthropic’s data collection. Anthropic’s current crawler documentation names ClaudeBot, Claude-SearchBot and Claude-User instead, and live requests that identify themselves as anthropic-ai are rare.
| Operator | Anthropic |
|---|---|
| Purpose | training · Anthropic (legacy) |
| robots.txt token | anthropic-ai |
| In the user-agent | Never sent: this token is only read from robots.txt |
| CrawlRadar weight | 1 of the crawler-access weights, never flagged critical |
| Vendor docs | Anthropic: Does Anthropic crawl data from the web? |
How to allow anthropic-ai in robots.txt
Give anthropic-ai its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: anthropic-ai
Allow: /How to block anthropic-ai in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: anthropic-ai
Disallow: /- Tokens are matched case-insensitively, but writing
anthropic-aiexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block anthropic-ai without any line in robots.txt.
Should you block anthropic-ai?
Keep its rule identical to your ClaudeBot rule. A file that disallows anthropic-ai but allows ClaudeBot sends a mixed signal nobody intended, and many robots.txt block-list generators still add it. Removing it changes nothing for Anthropic’s current crawlers.
How to verify a request is really anthropic-ai
There is no reliable way: Anthropic publishes no IP ranges for this retired token. A request claiming to be anthropic-ai could be anyone, so CrawlRadar records such hits as unverified rather than crediting them to Anthropic.
Anthropic’s other AI crawlers
Each Anthropic token is a separate robots.txt decision. Blocking one has no effect on the others.
| Token | Purpose | What blocking it costs |
|---|---|---|
ClaudeBot | training | Blocking it keeps your future pages out of Anthropic’s training data. Claude’s search citations come from Claude-SearchBot, which is unaffected. |
Claude-SearchBot | search index | This is the crawler that indexes pages for Claude’s search answers. Blocking it can remove you from the results Claude cites. |
Claude-User | live answer fetch | It fetches your page when someone asks Claude a question that needs it. Blocking it means Claude cannot retrieve your page for that person. |
Is anthropic-ai visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each anthropic-ai visit, the page it fetched, and whether it came from Anthropic. To check by hand, the guide to tracking AI crawlers has the log commands and each vendor’s IP lists. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
Do I still need anthropic-ai in robots.txt?
Not for Anthropic’s current crawlers, which identify as ClaudeBot, Claude-SearchBot and Claude-User. Leaving it in is harmless as long as its rule matches your ClaudeBot rule.Is a request claiming to be anthropic-ai genuine?
Probably not. There are no published IP ranges to check it against, so CrawlRadar records such hits as unverified rather than crediting them to Anthropic.