GPTBot: OpenAI's Training Crawler

What does GPTBot collect, and for what?

GPTBot collects publicly accessible web pages that OpenAI can use to train its generative AI foundation models. It does not index pages for ChatGPT search; OpenAI assigns that job to OAI-SearchBot. A robots.txt disallow for GPTBot tells OpenAI not to use the site's content for training.

A foundation model is the large general-purpose model behind products such as ChatGPT, and training data is the text it learns from before release. OpenAI describes GPTBot’s role in 2 sentences: it “is used to make our generative AI foundation models more useful and safe”, and it “is used to crawl content that may be used in training our generative AI foundation models”. The opt-out is defined just as plainly: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”

GPTBot is one of 4 agents on OpenAI’s crawler page. OAI-SearchBot fetches pages for ChatGPT search, ChatGPT-User opens pages that a user asks for, and OAI-AdsBot checks pages submitted as ads. Each has its own token, and OpenAI states that “each setting is independent of the others”.

What is GPTBot’s user-agent string?

GPTBot identifies itself with the user-agent string Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. OpenAI notes the version number changes over time. The robots.txt token is the short name GPTBot, which matches regardless of version.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

The user-agent string is the full identifier in each request and appears in server logs. The token is the part robots.txt matches: write User-agent: GPTBot, not the full string and not a version number. When GPTBot fetches robots.txt itself, OpenAI “may add a robots.txt marker to the user-agent string”, which separates those requests in logs that record no paths.

How do you block GPTBot in robots.txt?

Block GPTBot by adding a group for its token with a disallow rule: User-agent: GPTBot followed by Disallow: /. Replace / with a folder such as /members/ to block only that section. Rules written for other bots stay unchanged.

Each example below is a complete GPTBot group. Add it to the existing robots.txt at the root of every host the site uses.

Block the whole site

User-agent: GPTBot
Disallow: /

Outcome: GPTBot fetches no page, and the whole site carries the signal not to use its content for training.

Block one folder

User-agent: GPTBot
Disallow: /members/

Outcome: pages under /members/ are excluded; the rest of the site stays open to GPTBot.

Block everything except the blog

User-agent: GPTBot
Allow: /blog/
Disallow: /

Outcome: only pages under /blog/ stay available. The longer matching rule wins, so Allow: /blog/ overrides Disallow: / for blog URLs.

A group that names GPTBot also replaces the User-agent: * group for GPTBot, so any wildcard rule meant to apply to it belongs inside its own group too. OpenAI states that robots.txt updates take about 24 hours to reach its systems for search results; it publishes no separate figure for GPTBot. Run a GPTBot robots.txt test on the paths that matter to see the deciding line before the file goes live.

Does blocking GPTBot remove a site from ChatGPT?

No. Blocking GPTBot stops training use only; ChatGPT search appearance depends on OAI-SearchBot. OpenAI treats each robots.txt setting independently, so a site that disallows GPTBot and allows OAI-SearchBot stays eligible for ChatGPT search answers.

OpenAI gives this exact case as its example: a webmaster “can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot”. The rule set:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Outcome: no training use, full eligibility for ChatGPT search. What that second group controls, and what blocking it costs, is explained on the page about OpenAI’s search crawler.

How do you tell real GPTBot traffic from impostors?

Real GPTBot traffic comes from an IP address listed in OpenAI's file at openai.com/gptbot.json. Match each request claiming the GPTBot user agent against that list. Requests with the GPTBot string from other addresses are impostors and robots.txt does not govern them.

1. Filter the logsSelect requests whose user-agent string contains GPTBot/.
2. Take the source IPRead the client IP address of each request, not a forwarded header set by the client.
3. Compare with gptbot.jsonCheck whether the address falls inside a range in openai.com/gptbot.json.
4. Treat mismatches as impostorsHandle them with firewall rules, since a bot that fakes its name has no reason to read robots.txt.

How much does GPTBot crawl compared with the traffic it sends?

OpenAI's crawlers made 887 requests for every visitor they referred, according to Cloudflare's data for the first week of August 2025. In the same week, GPTBot and Anthropic's ClaudeBot together made up nearly half of the AI crawling Cloudflare observed.

Cloudflare published these figures on 28 August 2025 from its unfiltered network data. The 887-to-1 ratio covers all of OpenAI’s crawlers, not GPTBot alone. For comparison, Cloudflare put Anthropic’s ratio at nearly 50,000 to 1 and Perplexity’s at 118 to 1 in the same period, and found that training accounted for nearly 80% of AI bot crawling.

Those facts set up the owner’s decision.

What does a site gain or lose by blocking GPTBot?

Blocking GPTBot keeps a site's future content out of OpenAI's model training and removes GPTBot's crawl load, while ChatGPT search citations stay available through OAI-SearchBot. OpenAI's documentation does not address content collected before a block takes effect.

The decision is separate from search. A site can take either position on training and still keep its ChatGPT search visibility, which depends on OAI-SearchBot access and on the pages themselves.

Which crawlers do other AI companies use for training?

Each AI company names its own training crawler or token, and a GPTBot rule covers none of them. Anthropic runs ClaudeBot, which collects content that “could potentially contribute” to model training. Common Crawl runs CCBot, which builds an open repository of web crawl data. Meta runs Meta-ExternalAgent for “training foundation AI models or improving products”. Amazon runs Amazonbot, whose data “may be used to train Amazon AI models”. Apple and Google use tokens instead of separate crawlers: Applebot-Extended and Google-Extended decide whether content their existing crawlers fetch trains Apple’s and Google’s models.

A full training opt-out therefore names each token. The closest equivalent to GPTBot, with the same split between training and search, is Anthropic’s ClaudeBot.