By Iliass El Barhoumi · Published · Last updated
What is OAI-SearchBot?
OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT’s search results. OpenAI documents it separately from GPTBot so a site can appear in ChatGPT Search while still opting out of model training. Of OpenAI’s three tokens, it is the one that decides whether ChatGPT can link to your page in a search answer.
| Operator | OpenAI |
|---|---|
| Purpose | search index · ChatGPT Search |
| robots.txt token | OAI-SearchBot |
| In the user-agent | compatible; OAI-SearchBot/1.x; +https://openai.com/searchbot |
| CrawlRadar weight | 2 of the crawler-access weights, and a block is flagged critical |
| Vendor docs | OpenAI: Overview of OpenAI crawlers |
How to allow OAI-SearchBot in robots.txt
Give OAI-SearchBot its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: OAI-SearchBot
Allow: /How to block OAI-SearchBot in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: OAI-SearchBot
Disallow: /- Tokens are matched case-insensitively, but writing
OAI-SearchBotexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block OAI-SearchBot without any line in robots.txt.
Should you block OAI-SearchBot?
Allow OAI-SearchBot unless you never want ChatGPT to cite you. Blocking it is the most expensive single line a robots.txt can contain for ChatGPT visibility, which is why CrawlRadar weights it as a major crawler and flags a block as critical. OpenAI notes that a robots.txt change can take about 24 hours to reach its search systems.
How to verify a request is really OAI-SearchBot
OpenAI publishes the IP addresses OAI-SearchBot uses at https://openai.com/searchbot.json. A request that claims to be OAI-SearchBot from any other address is someone borrowing the name, and should be treated like any other scraper. CrawlRadar’s collector checks every hit against that list and labels it verified or spoofed.
OpenAI’s other AI crawlers
Each OpenAI token is a separate robots.txt decision. Blocking one has no effect on the others.
| Token | Purpose | What blocking it costs |
|---|---|---|
GPTBot | training | Blocking it keeps your pages out of future OpenAI model training. It does not remove you from ChatGPT Search, which crawls under OAI-SearchBot. |
ChatGPT-User | live answer fetch | It fetches your page while someone is waiting on an answer. Blocking it means ChatGPT may not open your page even when a user asks it to. |
Is OAI-SearchBot visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each OAI-SearchBot visit, the page it fetched, and whether it came from OpenAI. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
Can I block GPTBot and still appear in ChatGPT Search?
Yes. Allow OAI-SearchBot and disallow GPTBot. OpenAI documents the two as independent settings, one for search results and one for model training, and gives exactly this combination as its example.
Why does OAI-SearchBot never show up in Google Analytics?
Crawlers do not run JavaScript, so a tag-based analytics tool never records them. OAI-SearchBot only appears in your server logs or in a server-side collector such as CrawlRadar’s, which verifies each hit against openai.com/searchbot.json.