AI Crawler and robots.txt Guides

Use these guides to move from a policy question to a verified live robots.txt file. They cover crawler roles, training versus search access, syntax, deployment, testing, status codes, wildcards, and the limits of robots.txt.

Start with the decision you need to make

This guide library is organized around practical website-owner questions rather than a single universal bot list. Begin with the outcome: block model-training collection, remain visible in AI search, compare two agents from the same provider, troubleshoot a live file, or understand a robots.txt directive.

Recommended learning paths

  • Build an AI crawler policy: read how to block AI crawlers, then compare it with allowing AI search while blocking training.
  • Understand provider-specific controls: use the GPTBot/OAI-SearchBot, ClaudeBot/Claude-SearchBot, PerplexityBot/Perplexity-User, and Google-Extended/Googlebot comparisons.
  • Deploy and test: review the sitemap, HTTP status code, wildcard, and verification guides.
  • Avoid false expectations: read the guides on common mistakes and whether robots.txt stops AI scraping.
  • Apply a publishing policy: use the dedicated guidance for publishers and content-heavy sites.

A reliable workflow for every guide

  1. Identify the exact user-agent token and its documented purpose.
  2. Decide whether the desired result concerns crawling, search visibility, model-development use, or real access control.
  3. Change the smallest possible part of the active robots.txt source.
  4. Check the live HTTP response and test important URL paths.
  5. Use server logs and provider verification methods when identity or compliance matters.

Robots.txt is public, host-specific, and voluntary. The guides therefore distinguish policy signaling from authentication, indexing controls, and network enforcement. Use the glossary for concise definitions and the template library when you are ready to copy a starting rule.

Browse this section

Separate ClaudeBot training, Claude-SearchBot discovery, and secondary user-request access routes

ClaudeBot vs Claude-SearchBot: Training vs Search

ClaudeBot and Claude-SearchBot are separate Anthropic bots. ClaudeBot collects public web content that may contribute to model training, while Claude-SearchBot indexes content to improve Claude search results. You can disallow ClaudeBot while allowing Claude-SearchBot.

 Robots.txt workspace showing wrong location, overbroad blocking, path conflicts, and false security assumptions

Common robots.txt Mistakes and How to Fix Them

The most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.

 Voluntary robots.txt crawler rules shown separately from enforceable server-side access controls

Does robots.txt Stop AI Scraping? Limits and Better Controls

Robots.txt can stop compliant AI crawlers from requesting selected URLs, but it cannot enforce access against unidentified scrapers, spoofed user agents, browsers, or tools that ignore the protocol. Use precise crawler rules for policy, then add authentication, authorization, rate limits, or firewall controls where access must actually be prevented.

 Google Search crawling remaining open while a separate Gemini content-use route is disabled

Google-Extended vs Googlebot: Search and Gemini Control

Googlebot is the crawler used for Google Search and related search features. Google-Extended is a standalone robots.txt control token, not a separate HTTP crawler identity; it governs whether Google-crawled content may be used for Gemini model training and grounding. You can allow Googlebot while disallowing Google-Extended without opting out of Google Search.

 Separate GPTBot training and OAI-SearchBot search routes with independent website access controls

GPTBot vs OAI-SearchBot: Training vs Search

GPTBot and OAI-SearchBot are separate OpenAI crawler controls. GPTBot crawls content that may be used to improve and train OpenAI’s generative AI foundation models; OAI-SearchBot discovers pages for ChatGPT search results. You can block one and allow the other.

 Robots.txt pointing crawlers to a sitemap index while crawl permissions remain independent

How to Add a Sitemap to robots.txt Correctly

Add a standalone Sitemap line containing the full absolute URL of your XML sitemap or sitemap index, such as Sitemap: https://example.com/sitemap.xml. The line may appear anywhere in robots.txt and may be repeated, but it only helps discovery: it does not override Disallow rules or guarantee crawling or indexing.

 Website allowing an AI search discovery route while blocking a separate model-training route

How to Allow AI Search but Block AI Training

Use separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.

 Website selectively blocking AI training crawler routes while keeping search discovery routes open

How to Block AI Crawlers with robots.txt

To block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.

 Diagnostic flow checking a crawler token, URL rule match, server response, and access logs

How to Check Whether AI Crawlers Are Blocked

Fetch the live robots.txt file, select the crawler’s exact product token, and test the specific URL against the matching group and longest Allow or Disallow rule. Then check HTTP responses, caching, firewall rules, and logs, because a crawler may be allowed by robots.txt yet blocked by the server—or disallowed by robots.txt while an unidentified scraper still reaches the page.

 Automatic Perplexity search crawler compared with a single user-triggered page request

PerplexityBot vs Perplexity-User: What Changes?

PerplexityBot automatically discovers pages for Perplexity search and is governed by its own robots.txt token. Perplexity-User fetches a page after a person asks a question; Perplexity says this user-triggered agent generally ignores robots.txt. Use precise crawler rules for search visibility, and use server-side controls when access must be enforced.

 Public pages moving through AI search discovery and technical access toward possible source citation

robots.txt for AI Search Visibility and Citations

To support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.

 Website blocking selected AI training and dataset routes while keeping search discovery open

robots.txt for AI Training: Build a Selective Opt-Out

To restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.

 Publisher website allowing search discovery, blocking selected AI training access, and protecting subscriber content with server controls

robots.txt for Publishers and Content Sites

Publishers should use exact crawler groups instead of a blanket block: keep ordinary search crawling open, decide separately on AI training and AI-search access, and protect subscriber-only material with authentication rather than robots.txt.

 Crawler receiving successful, redirected, missing, and temporary-failure responses for robots.txt

robots.txt HTTP Status Codes: 200, 3xx, 4xx, and 5xx

A successful 2xx response lets crawlers parse robots.txt. Redirects may be followed, ordinary 4xx responses often mean no usable robots.txt restrictions, while 5xx and network failures are treated as temporary retrieval problems. Exact behavior and cache timing vary by crawler, so return a stable 200 response for a valid file and never use 401 or 403 as a substitute for crawl rules.

 URL paths passing through wildcard, exact-ending, and longest-match robots.txt rule selection

robots.txt Wildcards: * and $ Pattern Matching

In robots.txt, * matches zero or more characters and $ anchors the match to the end of the URL pattern. These are limited pattern operators, not full regular expressions. Crawlers compare applicable Allow and Disallow paths and use the most specific match; if equally specific rules conflict, Allow should win.

FAQ

Which guide should I read first?

Start with the guide that matches your goal: blocking AI crawlers, allowing AI search while blocking training, or checking whether a live rule works.

Are the comparison guides only about bot names?

No. They explain why providers publish separate tokens and how blocking one purpose can affect training, search visibility, or user-directed retrieval differently.

Do I need technical server access to use the guides?

Not always. Many changes can be made through a CMS or platform, but verification may require checking the live response, headers, logs, or hosting configuration.

Can the guides guarantee that every crawler will comply?

No. They explain robots.txt behavior for compliant crawlers and identify cases where server-side controls are required.

How often should I recheck my robots.txt policy?

Review it after provider documentation changes, platform or plugin updates, domain migrations, major content-policy changes, and unexpected crawler traffic.