By Iliass El Barhoumi · Published · Last updated
What is PerplexityBot?
PerplexityBot is the crawler Perplexity uses to surface and link websites in its search results. Perplexity’s documentation states that it is not used to crawl content for AI foundation models, and recommends allowing it, and its published IP ranges, for a site to appear in results.
| Operator | Perplexity |
|---|---|
| Purpose | search index · Perplexity search |
| robots.txt token | PerplexityBot |
| In the user-agent | compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot |
| CrawlRadar weight | 2 of the crawler-access weights, and a block is flagged critical |
| Vendor docs | Perplexity: Perplexity crawlers |
How to allow PerplexityBot in robots.txt
Give PerplexityBot its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: PerplexityBot
Allow: /How to block PerplexityBot in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: PerplexityBot
Disallow: /- Tokens are matched case-insensitively, but writing
PerplexityBotexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block PerplexityBot without any line in robots.txt.
Should you block PerplexityBot?
Allow PerplexityBot if you want Perplexity to cite you. Perplexity shows its sources next to every answer, so a page in its index is a referral channel you can measure. Perplexity says a robots.txt change can take up to 24 hours to take effect.
How to verify a request is really PerplexityBot
Perplexity publishes the IP addresses PerplexityBot uses at https://www.perplexity.com/perplexitybot.json. A request that claims to be PerplexityBot from any other address is someone borrowing the name, and should be treated like any other scraper. CrawlRadar’s collector checks every hit against that list and labels it verified or spoofed.
Perplexity’s other AI crawlers
Each Perplexity token is a separate robots.txt decision. Blocking one has no effect on the others.
| Token | Purpose | What blocking it costs |
|---|---|---|
Perplexity-User | live answer fetch | It fetches your page in response to a live question. Blocking it costs you the highest-intent visit a crawler can make. |
Is PerplexityBot visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each PerplexityBot visit, the page it fetched, and whether it came from Perplexity. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
Does PerplexityBot train AI models?
No, according to Perplexity: its documentation says PerplexityBot surfaces and links websites in search results and is not used to crawl content for AI foundation models.
How do I check whether PerplexityBot is crawling my site?
Match the PerplexityBot hits in your server logs against the ranges Perplexity publishes at perplexity.com/perplexitybot.json, or install CrawlRadar’s collector, which does the matching and lists each visit as verified or spoofed.