AI Crawler Checker

The checker needs JavaScript. The table of 16 crawlers below works without it.

Can AI bots crawl my site?

Yes, by default: every AI crawler is permitted to fetch a page unless the site's robots.txt disallows that path for the crawler's token or for User-agent: *. A firewall or bot-management rule can still block the request. User-triggered fetchers such as ChatGPT-User and Perplexity-User do not reliably follow robots.txt.

An AI crawler is a bot that an AI company runs to collect training data, build a search index or fetch a page for one user’s question. Each one names itself with a user-agent token, and robots.txt is the plain-text file at the root of a host that tells crawlers which paths they may request. A page reaches an AI answer only when 3 conditions hold:

1. Permission: robots.txt allows the path for that crawler’s token.
2. Delivery: the server, CDN and firewall answer the crawler’s request with the page.
3. Compliance: the crawler follows robots.txt, which training and search crawlers document and user-triggered fetchers do not always do.

The checker tests the first condition for all 16 agents at once.

How does the checker decide Allowed or Blocked?

The checker applies Google's published robots.txt rules. It picks the group with the most specific matching user agent, falls back to User-agent: *, then applies the longest matching Allow or Disallow path. On an equal-length tie, Allow wins. The deciding line is shown for every crawler.

Three precedence rules, from Google’s documentation and RFC 9309, produce every result:

1. Choose the groupA crawler obeys the group that names its own token and ignores every other group, including *. Groups naming the same token are combined. Tokens match without regard to case, and a value such as GPTBot/1.4 counts as GPTBot.
2. Fall back to the wildcardA crawler with no group of its own obeys the User-agent: * group. A file with neither leaves the crawler unrestricted.
3. Match the longest pathWithin the chosen group, the longest Allow or Disallow path that matches the tested path decides. * matches any characters and $ marks the end of a URL. An empty Disallow: line blocks nothing.

The example file pre-filled in the form shows rule 1 at work. At the path /, GPTBot and CCBot are blocked by their own Disallow: / lines. PerplexityBot is allowed, not by Allow: / in the * group but because its own group disallows only /private/. Test /private/report and the summary changes to “3 of 16 crawlers are blocked from /private/report: GPTBot, PerplexityBot, CCBot.”

AI operators publish no robots.txt error handling of their own, so this page states Google’s. For Amzn-SearchBot without its own group, the result shows the * outcome plus Amazon’s note that the bot follows rules given to other search bots. robots.txt also applies per host, so www.example.com and example.com each need a check.

What happens when robots.txt is missing or returns an error?

The HTTP status of robots.txt decides the result before any rule is read. Fetch mode reports the status it received.

Only a 2xx file carries rules, and a 404 allows every crawler. A 5xx answer costs most, because Google pauses crawling of the whole site.

Which 16 user agents does the checker test?

The checker tests 16 user agents from 9 operators: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Common Crawl, Meta and Amazon. They fall into 3 roles: training crawlers, search crawlers and user-triggered fetchers, plus 2 control tokens, Google-Extended and Applebot-Extended, that no bot sends as a user agent.

A training crawler collects content for future AI models. A search crawler indexes pages that an engine cites in live answers. A user-triggered fetcher opens one page because a user asked for it. A control token is a robots.txt name that changes how content fetched by another crawler is used; it never appears in a request.

12 of the 16 agents follow robots.txt as their operators document it, 2 user-triggered fetchers do not reliably follow it, and 2 control tokens are rules without a crawler. The checker omits anthropic-ai and Claude-Web, which Anthropic’s current crawler article does not list, and Bytespider, which has no operator documentation. Purposes, user-agent strings and IP lists for each bot are in the list of AI crawler user agents.

For user-triggered fetchers, the checker’s result states what the file says, not what the fetcher does. OpenAI adds that ChatGPT-User is not used to decide whether content appears in ChatGPT search.

Which AI crawlers should a site allow to get cited?

A site that wants AI citations allows the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Amzn-SearchBot. These fetch pages for live answers. Blocking the training crawlers GPTBot, ClaudeBot and CCBot does not remove a site from those answers, because each operator controls search and training with separate tokens.

Meta-ExternalAgent and Amazonbot sit between the two lists: Meta and Amazon describe both training and product uses for them. OpenAI applies robots.txt changes to search results in about 24 hours.

Does blocking Google-Extended remove a site from AI Overviews?

No. Google-Extended controls whether Google uses crawled content for Gemini model training and grounding, and it does not affect Google Search. Google states that robots.txt rules for Googlebot are the control for AI Overviews and AI Mode.

Google-Extended is a product token, not a crawler: no request carries it, and Google crawls with its existing user agents. Google’s AI features documentation names the actual control:

Google’s crawler documentation adds that “Google-Extended does not impact a site’s inclusion in Google Search”.

A site that blocks Googlebot to stay out of AI Overviews leaves Google Search as well.

What can a robots.txt check not detect?

A robots.txt check cannot see firewall blocks, CDN bot-management rules, noindex or nosnippet directives, or content that only renders with JavaScript. Each of these stops a page from being cited while robots.txt reports "Allowed". A full audit tests them from the crawler's side.

• Firewall rules that return 403 or a challenge page to bot traffic block a crawler that robots.txt allows.
• Bot-management settings at a CDN can refuse AI crawlers by default, with no line in robots.txt.
• Meta robots and X-Robots-Tag directives such as noindex and nosnippet keep an allowed page out of answers.
• JavaScript rendering hides content from crawlers that read only the HTML response.

Fetch mode also reads robots.txt as this site’s checker received it, and a firewall can answer AI crawlers differently. The complete AI visibility audit starts with crawler access and tests each of these layers before it looks at prompts and citations.

Run the next check

• Build an llms.txt file

  Create, validate and download the file with the free llms.txt generator. It runs in your browser.

• In preparation

  robots.txt rules for AI crawlers, explained

  Full syntax, group order and ready rule sets for training, search and user-triggered bots.

• Test beyond robots.txt

  Firewall, rendering and indexing checks from the crawler's side, plus a citation baseline across 6 AI engines. Request an audit.