AI Crawlers: User Agents, Operators and Purposes

Which AI crawlers exist, and who runs them?

This directory lists 20 documented AI crawlers and tokens from 9 operators: OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Common Crawl and Microsoft. OpenAI runs 4, Anthropic and Amazon run 3 each, Perplexity, Google, Apple and Meta run 2 each, and Common Crawl and Microsoft run 1 each.

Each row takes its purpose and robots.txt behaviour from the operator’s own page, quoted where the operator hedges. Open a row’s details for the full user-agent string and the way to verify the bot.

The table splits into 7 search crawlers, 5 user-triggered fetchers, 5 training crawlers including Common Crawl’s open corpus, 2 control tokens and 1 ad-review bot. Meta’s crawler page, updated 21 May 2026, also documents Meta-WebIndexer, which Meta says helps it cite and link content in Meta AI responses; it is not yet in this table.

What are AI crawlers?

AI crawlers are automated clients that AI companies send to websites to collect training data, build search indexes or fetch a page for one user's question. Each identifies itself with a user-agent token, which site owners address in robots.txt to allow or refuse it.

Two identifiers matter for every bot. The user-agent string is the full text the bot sends with each request, such as CCBot/2.0 (https://commoncrawl.org/faq/). The token is the short name inside it, such as CCBot, and robots.txt groups match on the token. robots.txt is the plain-text file at the root of each host that lists which paths each token may fetch. Operators publish the token and, in most cases, an IP list or reverse DNS pattern so site owners can tell real traffic from copies. A crawler that fetches a page is the precondition for any AI answer citing it.

How do training crawlers, search crawlers and user-triggered fetchers differ?

Training crawlers gather content for future models, search crawlers index pages that engines cite in live answers, and user-triggered fetchers open one page because a user asked. Blocking a training crawler leaves search citations intact; blocking a search crawler removes a site from that engine's answers.

The split is visible in traffic. Cloudflare reported on 28 August 2025 that training accounted for nearly 80% of AI bot crawling it observed. In its unfiltered data for the first week of August 2025, Anthropic’s crawlers made nearly 50,000 requests for every visitor they referred, OpenAI’s 887 and Perplexity’s 118. Operators that run search and training as separate tokens let a site accept one and refuse the other.

What is a control token such as Google-Extended?

A control token is a robots.txt name that no bot sends. It changes how content fetched by another crawler is used. The directory lists 2:

• Google-Extended decides whether content that Google’s existing crawlers fetch trains Gemini models and grounds Gemini Apps and Vertex AI. Google states that it “does not impact a site’s inclusion in Google Search”. For AI Overviews and AI Mode, Google names robots.txt rules for Googlebot as the control.
• Applebot-Extended decides whether content that Applebot fetches trains Apple’s foundation models. Apple states that “Applebot-Extended does not crawl webpages” and that pages disallowing it “can still be included in search results”.

Which AI crawlers ignore robots.txt?

Four documented agents do not reliably follow robots.txt: ChatGPT-User, Perplexity-User, Meta-ExternalFetcher and Amzn-User. Their operators state that user-initiated fetches fall outside robots.txt. Anthropic states the opposite for Claude-User: disallowing it stops Claude from retrieving the site for user queries.

• ChatGPT-User (OpenAI): “Because these actions are initiated by a user, robots.txt rules may not apply.”
• Perplexity-User (Perplexity): the fetcher “generally ignores robots.txt rules” because a user requested the page.
• Meta-ExternalFetcher (Meta): “this crawler may bypass robots.txt rules.”
• Amzn-User (Amazon): “it may not follow all robots.txt directives.”

Each of these fetchers acts for one person’s request, not an automatic crawl. OpenAI adds that ChatGPT-User “is not used to determine whether content may appear in Search”. A firewall rule on the operator’s published IP list refuses them where robots.txt does not.

How do you verify that a request comes from a real AI crawler?

Verify a crawler by matching the request's IP address against the operator's published IP list or by a reverse DNS lookup. OpenAI, Anthropic, Perplexity, Amazon, Google, Apple and Common Crawl publish IP lists; Google, Apple and Common Crawl also support reverse DNS. User-agent strings alone are spoofable.

Common Crawl states it is “aware of crawlers falsely identifying themselves as CCBot”, and Google warns that “the HTTP user agent string can be spoofed”. The method differs by operator:

A reverse DNS check runs in 2 steps: look up the hostname for the IP address, then confirm that the hostname resolves back to the same IP. Anthropic adds a warning against using its IP list to block: IP blocking “may not work correctly or persistently guarantee an opt-out”, because it stops its bots from reading robots.txt.

Which AI user agents are retired or undocumented?

anthropic-ai and Claude-Web are absent from Anthropic's current crawler documentation, which names only ClaudeBot, Claude-SearchBot and Claude-User. Bytespider appears in many block lists but has no operator documentation on record as of September 2026, so this directory excludes it.

• anthropic-ai: an older Anthropic name, not listed in the current article.
• Claude-Web: an older Anthropic name, not listed in the current article.
• Bytespider: a ByteDance user agent with no published operator documentation.

Rules for these names do no harm in robots.txt, but they control nothing that Anthropic documents today.

The list names each bot; a site’s robots.txt decides which of them get in.

Explore each crawler

Three crawlers and one file have their own pages, with rule sets, verification steps and the effect of blocking.

• GPTBot

  OpenAI's training crawler: user-agent string, IP list and rules to block it without leaving ChatGPT search.

• OAI-SearchBot

  The OpenAI crawler that decides whether pages appear as sources in ChatGPT search.

• ClaudeBot

  Anthropic's training crawler and its 2 siblings, with what blocking each one does to Claude answers.

• llms.txt proposal

  A file for agents, not a bot: format, v2 changes and what is documented about who reads it.