AI crawlers & traffic · 3 min read

# How to track AI crawlers on your website

To track AI crawlers, read your **server logs**, not your analytics: AI crawlers do not run JavaScript, so tag-based tools never see them. Match each request’s user agent against the known AI tokens, then verify the IP address against the ranges each vendor publishes, because the user agent alone is easy to fake.

By [Iliass El Barhoumi](/about) · Published 1 October 2026 · Last updated 1 October 2026

## Which AI crawlers should you look for?

Look for the tokens below, grouped by what the visit means. Search crawlers are the ones that decide citations; training crawlers only decide whether your pages feed future models. The [AI crawler directory](/crawlers) has a page for each.

| Purpose      | Tokens                                                                                                                                                       | A visit means                                  |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- |
| Search index | [OAI-SearchBot](/crawlers/oai-searchbot), [Claude-SearchBot](/crawlers/claude-searchbot), [PerplexityBot](/crawlers/perplexitybot)                           | The page can enter that engine’s citation pool |
| Live fetch   | [ChatGPT-User](/crawlers/chatgpt-user), [Claude-User](/crawlers/claude-user), [Perplexity-User](/crawlers/perplexity-user)                                   | Someone’s question needed this page right now  |
| Training     | [GPTBot](/crawlers/gptbot), [ClaudeBot](/crawlers/claudebot), [CCBot](/crawlers/ccbot), [Bytespider](/crawlers/bytespider), [Amazonbot](/crawlers/amazonbot) | The page may be used to train future models    |

Two common tokens never appear in logs: [Google-Extended](/crawlers/google-extended) and [Applebot-Extended](/crawlers/applebot-extended) are robots.txt permissions read by Google’s and Apple’s regular crawlers, not crawlers of their own.

## How to find AI crawlers in your server logs

Search the access log for the tokens, then count requests by crawler. On a server writing the common combined log format, this one command does both:

```
grep -oE "OAI-SearchBot|Claude-SearchBot|PerplexityBot|ChatGPT-User|Claude-User|Perplexity-User|GPTBot|ClaudeBot|CCBot|Bytespider|Amazonbot" access.log \
  | sort | uniq -c | sort -rn
```

To list the pages one crawler fetched, most requested first:

```
grep "OAI-SearchBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
```

On managed hosting without log access, the same data usually lives in your CDN or hosting provider’s request logs, which record the user agent of every request.

## How to verify a crawler is genuine

Check the request’s IP address against the vendor’s published list. A user agent is a claim; an IP inside the vendor’s range is evidence.

- OpenAI publishes a separate list per crawler: [openai.com/searchbot.json](https://openai.com/searchbot.json) for OAI-SearchBot, [openai.com/gptbot.json](https://openai.com/gptbot.json) for GPTBot and [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) for ChatGPT-User.
- Anthropic publishes one list for its crawlers at [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json).
- Perplexity publishes [perplexity.com/perplexitybot.json](https://www.perplexity.com/perplexitybot.json) for PerplexityBot and [perplexity.com/perplexity-user.json](https://www.perplexity.com/perplexity-user.json) for Perplexity-User.
- Amazon documents Amazonbot’s addresses and a reverse-DNS check under `crawl.amazonbot.amazon`.

Common Crawl and ByteDance publish no ranges, so CCBot and Bytespider visits can be counted but not verified.

## What to do with the numbers

Compare what the search crawlers fetch with what you want cited. A page in your sitemap that OAI-SearchBot, Claude-SearchBot and PerplexityBot have never fetched cannot be cited by those engines, however good it is. Check, in order:

1. Whether robots.txt blocks them, with the [AI crawler checker](/tools/ai-crawler-checker). The [crawler access check](/checks/crawler-access) explains how each rule is read.
2. Whether a CDN or firewall rule blocks them before robots.txt is read.
3. Whether the page is linked from pages they already crawl, and listed in your sitemap.

## The shortcut

CrawlRadar’s collector does all of the above continuously: it reads your own request path, identifies the AI crawlers, verifies each against the vendor’s published ranges, and shows which pages every crawler fetched next to the AI referral traffic those pages receive. [Add your website](/dashboard/new) to set it up in about two minutes, with a snippet for a Cloudflare Worker, Next.js middleware or a Node server, or the WordPress plugin. The [live demo](/demo) is CrawlRadar’s own dashboard, crawls included.

Crawls are half of the picture. The visitors AI assistants send back are the other half, and they arrive through your analytics, not your logs: see [how to track ChatGPT referral traffic in GA4](/blog/track-chatgpt-referral-traffic-ga4).

## Sources

- OpenAI, [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots).
- Anthropic, [Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- Perplexity, [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers).
- Google, [Google’s common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers): why Google-Extended never appears in logs.

## Frequently asked questions

- ### Why can I not see GPTBot or ClaudeBot in Google Analytics?  
Google Analytics only counts visits that execute its JavaScript tag, and AI crawlers fetch the HTML without running scripts. Their requests only exist in your server, CDN or hosting logs.
- ### How often do AI crawlers visit a website?  
It varies by site, crawler and how often the content changes, from daily passes on busy sites to a handful of fetches a month on small ones. The only reliable number is your own log, grouped by crawler and by page.
- ### Can a bot pretend to be GPTBot?  
Yes. Anyone can send “GPTBot” in a user-agent header. That is why OpenAI, Anthropic and Perplexity publish the IP ranges their crawlers use: a request is genuine only if its address falls inside the vendor’s list.

![Iliass El Barhoumi](/brand/founder-avatar.jpg)

Written by

[Iliass El Barhoumi](/about), Founder of CrawlRadar

CrawlRadar records the AI crawlers that visit a website and the visitors AI assistants send it. Every vendor claim in this guide links to the vendor’s own documentation.

## Related guides

Related guides answer the next question, same topic first.

- [AI crawlers & trafficHow to track ChatGPT referral traffic in GA4ChatGPT visits appear in GA4 with the source chatgpt.com. How to filter for them, give AI assistants their own channel, and what GA4 still cannot see.3 min read · Updated 1 October 2026](/blog/track-chatgpt-referral-traffic-ga4)
- [Getting citedWhat is generative engine optimization (GEO)?GEO is optimising a page to be quoted inside an AI answer rather than ranked as a link. What changes, what carries over from SEO, and what to do first.3 min read · Updated 1 October 2026](/blog/what-is-generative-engine-optimization)
- [Getting citedHow to get cited by ChatGPTFour things decide whether ChatGPT can cite your page: OAI-SearchBot access, server-rendered HTML, a clear publisher, and an answer-first passage.4 min read · Updated 1 October 2026](/blog/how-to-get-cited-by-chatgpt)

---

Source: https://www.crawlradar.xyz/blog/how-to-track-ai-crawlers
