By Iliass El Barhoumi · Published · Last updated
What is Claude-User?
Claude-User is the user agent Claude sends when a person asks a question that needs a live web page, or shares a link for Claude to read. Anthropic describes it as supporting Claude users directly: each request comes from a conversation, not from a scheduled crawl.
| Operator | Anthropic |
|---|---|
| Purpose | live answer fetch · Claude conversations |
| robots.txt token | Claude-User |
| In the user-agent | Claude-User |
| CrawlRadar weight | 1 of the crawler-access weights, never flagged critical |
| Vendor docs | Anthropic: Does Anthropic crawl data from the web? |
How to allow Claude-User in robots.txt
Give Claude-User its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: Claude-User
Allow: /How to block Claude-User in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: Claude-User
Disallow: /- Tokens are matched case-insensitively, but writing
Claude-Userexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block Claude-User without any line in robots.txt.
Should you block Claude-User?
Leave Claude-User allowed. Anthropic’s documentation says disabling it prevents Claude from retrieving your content in response to a user’s query, which may reduce your visibility for user-directed web search. A block means Claude cannot open your page even when someone asks about it by name.
How to verify a request is really Claude-User
Anthropic publishes the IP addresses Claude-User uses at https://claude.com/crawling/bots.json. A request that claims to be Claude-User from any other address is someone borrowing the name, and should be treated like any other scraper. CrawlRadar’s collector checks every hit against that list and labels it verified or spoofed.
Anthropic’s other AI crawlers
Each Anthropic token is a separate robots.txt decision. Blocking one has no effect on the others.
| Token | Purpose | What blocking it costs |
|---|---|---|
ClaudeBot | training | Blocking it keeps your future pages out of Anthropic’s training data. Claude’s search citations come from Claude-SearchBot, which is unaffected. |
Claude-SearchBot | search index | This is the crawler that indexes pages for Claude’s search answers. Blocking it can remove you from the results Claude cites. |
anthropic-ai | training | A legacy Anthropic token. Worth keeping in your robots.txt alongside ClaudeBot so an older group cannot contradict a newer one. |
Is Claude-User visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each Claude-User visit, the page it fetched, and whether it came from Anthropic. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
Does Claude-User follow robots.txt?
Yes. Anthropic documents Claude-User as a robots.txt token site owners can use to control which sites user-initiated requests can reach, and says its bots honour robots.txt directives.
Will Claude-User visits show up in my analytics?
No. Like every AI fetcher it does not run JavaScript, so an analytics tag never fires. It shows up in server logs and in CrawlRadar’s collector, which reports it separately from Claude-SearchBot and ClaudeBot.