Live demo: AI crawler and AI traffic dashboard
CrawlRadar’s own dashboard for crawlradar.xyz — real data, refreshed every few minutes.
AI crawlers
Crawls
789
Pages crawled
289
89%
verified
Verification
- Verified702 · 89%Came from an address the vendor publishes for its crawler.Verified70289%
- Unchecked75 · 10%The vendor publishes no way to check, or the check could not be completed.Unchecked7510%
- Spoofed12 · 2%Claimed to be a known crawler from an address that vendor does not crawl from.Spoofed122%
6 verified · 8 spoofed · 2 unchecked
By crawler
Most-crawled pagesAI crawlers only. Search engines, SEO tools and CrawlRadar scans are counted under “Outside the AI funnel”.
- /robots.txt315AI answers: 1, Indexing: 79, Training: 220, Unclassified: 15
- /sitemap.xml207Training: 207
- /45AI answers: 9, Indexing: 18, Training: 5, Unclassified: 13
- /llms.txt11Indexing: 9, Training: 2
- /pricing8AI answers: 2, Indexing: 4, Training: 2
- /contact8Indexing: 7, Training: 1
- /blog/how-to-track-ai-crawlers7Indexing: 7
- /blog6AI answers: 1, Indexing: 4, Training: 1
- /demo6AI answers: 1, Indexing: 4, Training: 1
- /blog/track-chatgpt-referral-traffic-ga46Indexing: 6
Engine coverageWhich answer engines fetched your pages this period, and what for. An empty row is an engine that can’t cite you yet. Google’s row covers NotebookLM and Vertex grounding; Googlebot is listed under general search.
| Engine | Answer fetches | Index crawls | Training crawls |
|---|---|---|---|
| ChatGPT | 7 | 20 | none |
| 1 | none | none | |
| Perplexity | none | 20 | none |
| Claude | 6 | none | 415 |
| Bing | none | 15 | none |
| Grok | none | none | none |
| Amazon | none | 173 | none |
| Semrush | none | none | none |
| Majestic | none | none | none |
| Ahrefs | none | none | none |
| Apple | none | 50 | none |
| Common Crawl | none | none | 39 |
| Moz | none | none | none |
| Meta AI | none | none | none |
| ByteDance | none | none | 9 |
| Other | none | none | none |
| Baidu | none | 4 | none |
| DuckDuckGo | 2 | none | none |
Stray URLsURLs crawlers requested that your sitemap doesn’t account for. Not on your site: the URL fails to load — redirect it or let it 404. Not in sitemap: the page exists, but your sitemap doesn’t list it. The busiest are checked live on your site.
Checking your sitemap…
Outside the AI funnelEverything your collector saw that isn’t a citation signal. Counted here, never in the AI numbers above.
General search
Search engine crawlers like Googlebot and Bingbot. Google’s AI Overviews draw on this crawl, but no request shows which fetch fed an answer — so it’s counted, not credited.87SEO tools
Backlink and SEO crawlers. They index the web for their own products — no bearing on AI citations, but they do use crawl budget.363Unrecognised
Crawlers we couldn’t match to a known bot. They’re counted only here — nothing is guessed. Once identified, they move to the right group for every site.5,248AI traffic
Read by AI answersTimes an assistant — ChatGPT, Claude, Perplexity… — fetched one of your pages to answer someone’s question. Counted by the collector, whether or not anyone clicked.
16
4 assistants
Visits from AIVisits that arrived from an AI assistant’s answer, counted by the tag. Many assistants hide where a visit came from, so this is a floor, not a total.
0
of 300 visits tracked
Click-throughVisits from AI per answer that read your pages. Approximate: an assistant can send a visit without a live fetch, and many answers show no link at all.
0%
of answers led to a visit
This is crawlradar.xyz. See the same dashboard for your site.
Add your website