Common Crawl · training
Is CCBot blocked on your site?
CCBot is Common Crawl’s training token for Common Crawl, and whether it reaches you is decided by one line in your robots.txt. Common Crawl’s archive feeds many models indirectly. Blocking it is a broad opt-out with a long tail.
- Secondary crawler weight
- CCBot robots.txt token
Body copy pending
Phase 2 adds the parts that rank: the exact allow and block snippets for CCBot, what Common Crawl publishes about it, the CDN and plugin defaults that block it without telling you, and how it differs from the vendor’s other tokens.
This page is noindex until that lands, and it is not in the sitemap.