Product Hunt launch · 20% off your first 3 months withSee pricing →

AI crawler directory

AI crawler list: all 14 AI bots and what each one does

These are the 14 AI crawler tokens CrawlRadar reads your robots.txt for, one group at a time, for the exact path scanned — and only 3 of them (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the search indexes answer engines cite from. The rest train models or fetch pages on request.

Check my robots.txt
  • 14 tokens checked
  • 3 search crawlers

By · Published · Last updated

AI crawlers fall into four groups, and only one of them decides citations. Search crawlers build the indexes answer engines quote from; live fetchers open a page because someone asked; training crawlers and policy tokens decide whether your pages feed future models.

Policy tokens: read from robots.txt, never sent

No request carries these names. They control how the vendor’s regular crawl may be used.

FAQ

Frequently asked questions

What each kind of AI crawler does, and which ones decide whether you get cited.

  • Which AI crawlers should I allow in robots.txt?

    Allow the search crawlers, OAI-SearchBot, Claude-SearchBot and PerplexityBot, if you want ChatGPT, Claude and Perplexity to cite you. Allowing or blocking the training crawlers, such as GPTBot, ClaudeBot and CCBot, is a separate choice about model training that does not affect citations.
  • Is it safe to block GPTBot?

    Yes, if you do not want your pages used to train OpenAI’s models. OpenAI documents GPTBot as a training crawler and OAI-SearchBot as the one behind ChatGPT Search, and states the two settings are independent.
  • Does Google-Extended control Google AI Overviews?

    No. Google-Extended governs Gemini training and grounding, and Google states it does not affect inclusion in Google Search. AI Overviews are a Search feature built from Googlebot’s crawl.

Can AI find your website?

See which AI crawlers visit, the pages they fetch, and how many visitors ChatGPT, Perplexity, Gemini and Claude send you.

14-day free trial.No credit card.