How to track AI crawlers on your website
To track AI crawlers, read your server logs, not your analytics: AI crawlers do not run JavaScript, so tag-based tools never see them. Match each request’s user agent against the known AI tokens, then verify the IP address against the ranges each vendor publishes, because the user agent alone is easy to fake.
By Iliass El Barhoumi · Published · Last updated
Which AI crawlers should you look for?
Look for the tokens below, grouped by what the visit means. Search crawlers are the ones that decide citations; training crawlers only decide whether your pages feed future models. The AI crawler directory has a page for each.
| Purpose | Tokens | A visit means |
|---|---|---|
| Search index | OAI-SearchBot, Claude-SearchBot, PerplexityBot | The page can enter that engine’s citation pool |
| Live fetch | ChatGPT-User, Claude-User, Perplexity-User | Someone’s question needed this page right now |
| Training | GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot | The page may be used to train future models |
Two common tokens never appear in logs: Google-Extended and Applebot-Extended are robots.txt permissions read by Google’s and Apple’s regular crawlers, not crawlers of their own.
How to find AI crawlers in your server logs
Search the access log for the tokens, then count requests by crawler. On a server writing the common combined log format, this one command does both:
grep -oE "OAI-SearchBot|Claude-SearchBot|PerplexityBot|ChatGPT-User|Claude-User|Perplexity-User|GPTBot|ClaudeBot|CCBot|Bytespider|Amazonbot" access.log \
| sort | uniq -c | sort -rnTo list the pages one crawler fetched, most requested first:
grep "OAI-SearchBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20On managed hosting without log access, the same data usually lives in your CDN or hosting provider’s request logs, which record the user agent of every request.
How to verify a crawler is genuine
Check the request’s IP address against the vendor’s published list. A user agent is a claim; an IP inside the vendor’s range is evidence.
- OpenAI publishes separate lists for OAI-SearchBot, GPTBot and ChatGPT-User.
- Anthropic publishes one list for its crawlers at claude.com/crawling/bots.json.
- Perplexity publishes lists for PerplexityBot and Perplexity-User.
- Amazon documents Amazonbot’s addresses and a reverse-DNS check under
crawl.amazonbot.amazon.
Common Crawl and ByteDance publish no ranges, so CCBot and Bytespider visits can be counted but not verified.
What to do with the numbers
Compare what the search crawlers fetch with what you want cited. A page in your sitemap that OAI-SearchBot, Claude-SearchBot and PerplexityBot have never fetched cannot be cited by those engines, however good it is. Check, in order:
- Whether robots.txt blocks them, with the AI crawler checker.
- Whether a CDN or firewall rule blocks them before robots.txt is read.
- Whether the page is linked from pages they already crawl, and listed in your sitemap.
The shortcut
CrawlRadar’s collector does all of the above continuously: it reads your own request path, identifies the AI crawlers, verifies each against the vendor’s published ranges, and shows which pages every crawler fetched next to the AI referral traffic those pages receive. Add your website to set it up in about two minutes, with a snippet for a Cloudflare Worker, Next.js middleware or a Node server, or the WordPress plugin.
Sources
- OpenAI, Overview of OpenAI crawlers.
- Anthropic, Does Anthropic crawl data from the web?
- Perplexity, Perplexity crawlers.
- Google, Google’s common crawlers: why Google-Extended never appears in logs.
Frequently asked questions
Why can I not see GPTBot or ClaudeBot in Google Analytics?
Google Analytics only counts visits that execute its JavaScript tag, and AI crawlers fetch the HTML without running scripts. Their requests only exist in your server, CDN or hosting logs.
How often do AI crawlers visit a website?
It varies by site, crawler and how often the content changes, from daily passes on busy sites to a handful of fetches a month on small ones. The only reliable number is your own log, grouped by crawler and by page.
Can a bot pretend to be GPTBot?
Yes. Anyone can send “GPTBot” in a user-agent header. That is why OpenAI, Anthropic and Perplexity publish the IP ranges their crawlers use: a request is genuine only if its address falls inside the vendor’s list.