By Iliass El Barhoumi · Published · Last updated
What is anthropic-ai?
anthropic-ai is an older robots.txt token associated with Anthropic’s data collection. Anthropic’s current crawler documentation names ClaudeBot, Claude-SearchBot and Claude-User instead, and live requests that identify themselves as anthropic-ai are rare.
| Operator | Anthropic |
|---|---|
| Purpose | training · Anthropic (legacy) |
| robots.txt token | anthropic-ai |
| In the user-agent | Never sent: this token is only read from robots.txt |
| CrawlRadar weight | 1 of the crawler-access weights, never flagged critical |
| Vendor docs | Anthropic: Does Anthropic crawl data from the web? |
How to allow anthropic-ai in robots.txt
Give anthropic-ai its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: anthropic-ai
Allow: /How to block anthropic-ai in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: anthropic-ai
Disallow: /- Tokens are matched case-insensitively, but writing
anthropic-aiexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block anthropic-ai without any line in robots.txt.
Should you block anthropic-ai?
Keep its rule identical to your ClaudeBot rule. A file that disallows anthropic-ai but allows ClaudeBot sends a mixed signal nobody intended, and many robots.txt block-list generators still add it. Removing it changes nothing for Anthropic’s current crawlers.
How to verify a request is really anthropic-ai
There is no reliable way: Anthropic publishes no IP ranges for this retired token. A request claiming to be anthropic-ai could be anyone, so CrawlRadar records such hits as unverified rather than crediting them to Anthropic.
Anthropic’s other AI crawlers
Each Anthropic token is a separate robots.txt decision. Blocking one has no effect on the others.
| Token | Purpose | What blocking it costs |
|---|---|---|
ClaudeBot | training | Blocking it keeps your future pages out of Anthropic’s training data. Claude’s search citations come from Claude-SearchBot, which is unaffected. |
Claude-SearchBot | search index | This is the crawler that indexes pages for Claude’s search answers. Blocking it can remove you from the results Claude cites. |
Claude-User | live answer fetch | It fetches your page when someone asks Claude a question that needs it. Blocking it means Claude cannot retrieve your page for that person. |
Is anthropic-ai visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each anthropic-ai visit, the page it fetched, and whether it came from Anthropic. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
Do I still need anthropic-ai in robots.txt?
Not for Anthropic’s current crawlers, which identify as ClaudeBot, Claude-SearchBot and Claude-User. Leaving it in is harmless as long as its rule matches your ClaudeBot rule.
Is a request claiming to be anthropic-ai genuine?
Probably not. There are no published IP ranges to check it against, so CrawlRadar records such hits as unverified rather than crediting them to Anthropic.