Product Hunt launch · 20% off your first 3 months withSee pricing →

AI crawlers & traffic · 3 min read

How to track AI crawlers on your website

To track AI crawlers, read your server logs, not your analytics: AI crawlers do not run JavaScript, so tag-based tools never see them. Match each request’s user agent against the known AI tokens, then verify the IP address against the ranges each vendor publishes, because the user agent alone is easy to fake.

By · Published · Last updated
On this page

Which AI crawlers should you look for?

Look for the tokens below, grouped by what the visit means. Search crawlers are the ones that decide citations; training crawlers only decide whether your pages feed future models. The AI crawler directory has a page for each.

AI crawler tokens grouped by purpose
PurposeTokensA visit means
Search indexOAI-SearchBot, Claude-SearchBot, PerplexityBotThe page can enter that engine’s citation pool
Live fetchChatGPT-User, Claude-User, Perplexity-UserSomeone’s question needed this page right now
TrainingGPTBot, ClaudeBot, CCBot, Bytespider, AmazonbotThe page may be used to train future models

Two common tokens never appear in logs: Google-Extended and Applebot-Extended are robots.txt permissions read by Google’s and Apple’s regular crawlers, not crawlers of their own.

How to find AI crawlers in your server logs

Search the access log for the tokens, then count requests by crawler. On a server writing the common combined log format, this one command does both:

grep -oE "OAI-SearchBot|Claude-SearchBot|PerplexityBot|ChatGPT-User|Claude-User|Perplexity-User|GPTBot|ClaudeBot|CCBot|Bytespider|Amazonbot" access.log \
  | sort | uniq -c | sort -rn

To list the pages one crawler fetched, most requested first:

grep "OAI-SearchBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

On managed hosting without log access, the same data usually lives in your CDN or hosting provider’s request logs, which record the user agent of every request.

How to verify a crawler is genuine

Check the request’s IP address against the vendor’s published list. A user agent is a claim; an IP inside the vendor’s range is evidence.

Common Crawl and ByteDance publish no ranges, so CCBot and Bytespider visits can be counted but not verified.

What to do with the numbers

Compare what the search crawlers fetch with what you want cited. A page in your sitemap that OAI-SearchBot, Claude-SearchBot and PerplexityBot have never fetched cannot be cited by those engines, however good it is. Check, in order:

  1. Whether robots.txt blocks them, with the AI crawler checker. The crawler access check explains how each rule is read.
  2. Whether a CDN or firewall rule blocks them before robots.txt is read.
  3. Whether the page is linked from pages they already crawl, and listed in your sitemap.

The shortcut

CrawlRadar’s collector does all of the above continuously: it reads your own request path, identifies the AI crawlers, verifies each against the vendor’s published ranges, and shows which pages every crawler fetched next to the AI referral traffic those pages receive. Add your website to set it up in about two minutes, with a snippet for a Cloudflare Worker, Next.js middleware or a Node server, or the WordPress plugin. The live demo is CrawlRadar’s own dashboard, crawls included.

Crawls are half of the picture. The visitors AI assistants send back are the other half, and they arrive through your analytics, not your logs: see how to track ChatGPT referral traffic in GA4.

Sources

Frequently asked questions

  • Why can I not see GPTBot or ClaudeBot in Google Analytics?

    Google Analytics only counts visits that execute its JavaScript tag, and AI crawlers fetch the HTML without running scripts. Their requests only exist in your server, CDN or hosting logs.
  • How often do AI crawlers visit a website?

    It varies by site, crawler and how often the content changes, from daily passes on busy sites to a handful of fetches a month on small ones. The only reliable number is your own log, grouped by crawler and by page.
  • Can a bot pretend to be GPTBot?

    Yes. Anyone can send “GPTBot” in a user-agent header. That is why OpenAI, Anthropic and Perplexity publish the IP ranges their crawlers use: a request is genuine only if its address falls inside the vendor’s list.
Iliass El Barhoumi

Written by

, Founder of CrawlRadar

CrawlRadar records the AI crawlers that visit a website and the visitors AI assistants send it. Every vendor claim in this guide links to the vendor’s own documentation.

Related guides

Related guides answer the next question, same topic first.

Can AI find your website?

See which AI crawlers visit, the pages they fetch, and how many visitors ChatGPT, Perplexity, Gemini and Claude send you.

14-day free trial.No credit card.