ClaudeBot: Anthropic's Training Crawler and Its Two Siblings

What is ClaudeBot?

ClaudeBot is the web crawler Anthropic uses to collect public web content that can contribute to training its Claude models. Its robots.txt token is ClaudeBot. Disallowing it tells Anthropic to exclude the site's future content from training datasets; it does not control Claude's search answers.

Anthropic is the AI company that builds Claude, a family of large language models and the assistant of the same name. Its crawler article describes ClaudeBot’s job in one sentence: it “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” The effect of a block is stated just as directly: restricting ClaudeBot “signals that the site’s future materials should be excluded from our AI model training datasets.”

ClaudeBot is not the bot behind Claude’s live answers. Anthropic assigns search indexing and user requests to 2 other tokens, and a ClaudeBot rule touches neither. OpenAI splits the same jobs between GPTBot and OAI-SearchBot; the page on OpenAI’s GPTBot equivalent shows the parallel rules.

What do Claude-SearchBot and Claude-User do?

Claude-SearchBot indexes web pages to improve the results Claude's search returns, and Claude-User fetches a page when a person asks Claude a question. Both are separate robots.txt tokens from ClaudeBot, so a site can allow them while refusing training.

• Claude-SearchBot “navigates the web to improve search result quality for users”. Anthropic says it analyses content “specifically to enhance the relevance and accuracy of search responses”.
• Claude-User acts for one person: “When individuals ask questions to Claude, it may access websites using a Claude-User agent.” Anthropic adds that the token “allows site owners to control which sites can be accessed through these user-initiated requests”.

Claude-User differs from some other companies’ user-triggered fetchers. Anthropic presents it as a token that site owners control through robots.txt, while OpenAI and Perplexity describe their equivalents as not reliably following robots.txt.

What happens when you block each Anthropic bot?

Blocking ClaudeBot removes future content from Anthropic's training data, blocking Claude-SearchBot stops Claude from indexing the site for search, and blocking Claude-User stops Claude from fetching pages for user questions. Only the last two reduce a site's presence in Claude's answers.

The matrix gives 3 independent choices. A site that wants to stay citable in Claude while refusing training blocks ClaudeBot and leaves the other 2 tokens allowed.

How do you write robots.txt rules for ClaudeBot?

Write a group that names the token and a disallow path: User-agent: ClaudeBot then Disallow: /. To slow ClaudeBot instead of blocking it, use Crawl-delay: 1, a non-standard directive that Anthropic supports. Add the same rules to robots.txt on every subdomain.

Block training, keep Claude answers

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Outcome: future content stays out of training data, and Claude can still index the site for search and fetch pages for users.

Slow ClaudeBot instead of blocking it

User-agent: ClaudeBot
Crawl-delay: 1

Outcome: ClaudeBot keeps crawling at a limited rate. Anthropic calls Crawl-delay “non-standard” and supports it “to limit crawling activity”. Apple and Amazon state that their crawlers do not follow it.

Block all three Anthropic bots

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

Outcome: no training use, no search indexing and no user fetches from Anthropic.

Run a ClaudeBot access test on the paths that matter before publishing: the checker reports ClaudeBot, Claude-SearchBot and Claude-User separately, with the rule that decides each one.

Do the rules need repeating on subdomains?

Yes. robots.txt applies to one host, and Anthropic asks site owners to add the rules “for every subdomain that you wish to opt out from”. A block in example.com/robots.txt does not cover blog.example.com or shop.example.com; each host serves its own file with its own groups.

Why does Anthropic advise against blocking ClaudeBot by IP address?

Anthropic warns that IP blocking prevents its bots from reading a site's robots.txt, so the opt-out is not guaranteed to hold. Robots.txt is the documented opt-out. Anthropic publishes its crawler IP ranges at claude.com/crawling/bots.json for verifying requests, not for blocking them.

Anthropic’s wording: blocking its IP addresses “may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” The article states that opting out “requires modifying the robots.txt file”. The IP list serves verification: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.” A request that claims to be ClaudeBot from an address outside bots.json does not come from Anthropic.

How much does Anthropic crawl for each visitor it sends?

Anthropic's crawlers made nearly 50,000 requests for every visitor they referred, in Cloudflare's unfiltered data for the first week of August 2025. OpenAI's ratio in the same week was 887 to 1 and Perplexity's was 118 to 1.

Cloudflare published the figures on 28 August 2025. In the same data, ClaudeBot and GPTBot together accounted for nearly half of the AI crawling Cloudflare observed, and training made up nearly 80% of AI bot crawling. The ratio compares all of Anthropic’s crawler requests with the visits Claude referred.

Old block lists often name tokens Anthropic no longer documents.

Are anthropic-ai and Claude-Web still needed in robots.txt?

No current Anthropic documentation names anthropic-ai or Claude-Web. Anthropic's crawler article, updated 7 April 2026, lists 3 tokens: ClaudeBot, Claude-SearchBot and Claude-User. Rules for the older names do no harm, but only the 3 current tokens are documented.

A robots.txt file that still carries only anthropic-ai or Claude-Web groups has no rule for the 3 documented tokens, so those bots follow the User-agent: * group instead. Adding groups for ClaudeBot, Claude-SearchBot and Claude-User makes the intended policy explicit.

Where do you report a misbehaving Anthropic crawler?

Anthropic asks site owners to email claudebot@anthropic.com when its bots appear to malfunction, and to write “from an email that includes the domain you are contacting us about”, since reports from other addresses are hard to verify. Include the affected host, the time range and the source IPs, checked against bots.json first. For the bots other AI companies run, the full AI crawler list gives each operator’s tokens and verification method.