OAI-SearchBot: OpenAI's Search Crawler

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's search crawler: it fetches web pages so they can appear as sources in ChatGPT's search features. Its robots.txt token is OAI-SearchBot. It does not collect model training data, which OpenAI assigns to a separate crawler, GPTBot.

OpenAI is the company that builds ChatGPT, and ChatGPT search is the part of ChatGPT that answers with linked web sources. OAI-SearchBot supplies those sources. OpenAI’s crawler page describes it in one line: “OAI-SearchBot is for search.” The same page lists 4 agents in total: OAI-SearchBot for search, GPTBot for training, ChatGPT-User for user actions and OAI-AdsBot for checking ad landing pages. OpenAI recommends allowing OAI-SearchBot in robots.txt “and allowing requests from our published IP ranges”, which extends the permission to firewalls and CDN rules.

What happens when a site blocks OAI-SearchBot?

A site that blocks OAI-SearchBot is not shown in ChatGPT search answers. OpenAI states that such a site can still appear as a navigational link. Blocking OAI-SearchBot has no effect on GPTBot, because OpenAI treats each robots.txt setting independently.

OpenAI’s exact words: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” A navigational link is a plain link to the site, for example when a user asks for the site by name, rather than a cited source inside an answer. The block therefore removes the page from the answer text and its source list while leaving the domain reachable by name.

The independence rule works in both directions. OpenAI writes that “each setting is independent of the others”, so a disallow for OAI-SearchBot says nothing about training. A site that wants neither search appearance nor training use needs 2 groups, one per token.

How does OAI-SearchBot differ from GPTBot and ChatGPT-User?

OAI-SearchBot indexes pages for ChatGPT search, GPTBot collects content for training OpenAI's foundation models, and ChatGPT-User opens a page when a user asks. Only OAI-SearchBot controls search appearance. ChatGPT-User is not an automatic crawler, and robots.txt rules do not reliably apply to it.

Only one of the 3 agents decides search appearance. OpenAI states that ChatGPT-User “is not used to determine whether content may appear in Search” and points site owners to OAI-SearchBot for search opt-outs. The GPTBot training crawler has its own page with its user-agent string, IP list and blocking examples.

A site that allows both OAI-SearchBot and GPTBot can be crawled once for both uses: OpenAI writes that “we may use the results from just one crawl for both use cases to avoid duplicative crawling.”

How do you allow OAI-SearchBot in robots.txt?

Allow OAI-SearchBot with a group that names its token and permits the paths you want in ChatGPT search: User-agent: OAI-SearchBot followed by Allow: /. OpenAI also asks sites to let its published IP ranges through any firewall.

A group that names OAI-SearchBot replaces the User-agent: * group for that crawler, so any * rule the site still wants applied has to be repeated inside it. Three common policies:

Policy A: search yes, training no

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Outcome: the site stays eligible for ChatGPT search answers, and GPTBot receives the signal not to use its content for training.

Policy B: allow all OpenAI crawling

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

Outcome: eligible for ChatGPT search and available for training; OpenAI can serve both uses from a single crawl.

Policy C: keep one folder out of ChatGPT search

User-agent: OAI-SearchBot
Disallow: /members/

Outcome: pages under /members/ are not shown in ChatGPT search answers, and every other path stays eligible.

Before publishing any of these, check OAI-SearchBot access for the paths that matter for all of them: the checker shows the line that decides each crawler’s result.

How long until ChatGPT search reflects a robots.txt change?

OpenAI gives one figure: “it can take ~24 hours from a site’s robots.txt update for our systems to adjust” for search results. OpenAI ties this figure to search results only and publishes no separate timing for GPTBot.

How can you confirm a request came from OAI-SearchBot?

Confirm OAI-SearchBot by checking the request's IP address against OpenAI's list at openai.com/searchbot.json. The user-agent string contains OAI-SearchBot/1.4 and +https://openai.com/searchbot, but any client can copy a user-agent string, so the IP match is the proof.

Normal page requests carry this string (OpenAI notes the version number may change):

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Requests for robots.txt itself can carry an extra robots.txt marker, which separates those fetches in logs that record no paths:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

Match the source IP of each request that claims either string against searchbot.json. A request with the string from an address outside the list is not OAI-SearchBot, and the same list is the one to allow in a firewall.

Access makes a page eligible. What makes ChatGPT pick it?

Does allowing OAI-SearchBot guarantee ChatGPT citations?

No. Allowing OAI-SearchBot makes pages eligible for ChatGPT search; it does not select them. ChatGPT chooses which eligible pages to cite for each answer. Access is the first condition, and content relevance decides the rest.

OpenAI’s crawler page covers access only; it does not describe how ChatGPT ranks or selects the sources it cites. Two access checks come first: robots.txt allows OAI-SearchBot on the pages that matter, and the firewall lets OpenAI’s published IP ranges through. After that, the work moves to the pages themselves, which passages answer the question, and how clearly the site is identified. The steps for that stage are in the guide to ranking in ChatGPT search.