AI Crawler and robots.txt Guides

Use these guides to move from a policy question to a verified live robots.txt file. They cover crawler roles, training versus search access, syntax, deployment, testing, status codes, wildcards, and the limits of robots.txt.

Start with the decision you need to make

This guide library is organized around practical website-owner questions rather than a single universal bot list. Begin with the outcome: block model-training collection, remain visible in AI search, compare two agents from the same provider, troubleshoot a live file, or understand a robots.txt directive.

Recommended learning paths

  • Build an AI crawler policy: read how to block AI crawlers, then compare it with allowing AI search while blocking training.
  • Understand provider-specific controls: use the GPTBot/OAI-SearchBot, ClaudeBot/Claude-SearchBot, PerplexityBot/Perplexity-User, and Google-Extended/Googlebot comparisons.
  • Deploy and test: review the sitemap, HTTP status code, wildcard, and verification guides.
  • Avoid false expectations: read the guides on common mistakes and whether robots.txt stops AI scraping.
  • Apply a publishing policy: use the dedicated guidance for publishers and content-heavy sites.

A reliable workflow for every guide

  1. Identify the exact user-agent token and its documented purpose.
  2. Decide whether the desired result concerns crawling, search visibility, model-development use, or real access control.
  3. Change the smallest possible part of the active robots.txt source.
  4. Check the live HTTP response and test important URL paths.
  5. Use server logs and provider verification methods when identity or compliance matters.

Robots.txt is public, host-specific, and voluntary. The guides therefore distinguish policy signaling from authentication, indexing controls, and network enforcement. Use the glossary for concise definitions and the template library when you are ready to copy a starting rule.

Browse this section

Separate ClaudeBot training, Claude-SearchBot discovery, and secondary user-request access routes

ClaudeBot vs Claude-SearchBot vs Claude-User

Anthropic uses three distinct agents: ClaudeBot for model-development crawling, Claude-SearchBot for search quality and indexing, and Claude-User for user-directed retrieval. Treat them as separate policy decisions.

 Robots.txt workspace showing wrong location, overbroad blocking, path conflicts, and false security assumptions

Common robots.txt Mistakes and How to Fix Them

The most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.

 Voluntary robots.txt crawler rules shown separately from enforceable server-side access controls

Does robots.txt Stop AI Scraping? Limits and Better Controls

Robots.txt can stop compliant AI crawlers from requesting selected URLs, but it cannot enforce access against unidentified scrapers, spoofed user agents, browsers, or tools that ignore the protocol. Use precise crawler rules for policy, then add authentication, authorization, rate limits, or firewall controls where access must actually be prevented.

 Google Search crawling remaining open while a separate Gemini content-use route is disabled

Google-Extended vs Googlebot: Search and Gemini Control

Googlebot is the crawler used for Google Search and related search features. Google-Extended is a standalone robots.txt control token, not a separate HTTP crawler identity; it governs whether Google-crawled content may be used for Gemini model training and grounding. You can allow Googlebot while disallowing Google-Extended without opting out of Google Search.

 Separate GPTBot training and OAI-SearchBot search routes with independent website access controls

GPTBot vs OAI-SearchBot: Training vs Search

GPTBot and OAI-SearchBot are separate OpenAI crawler controls. GPTBot crawls content that may be used to improve and train OpenAI’s generative AI foundation models; OAI-SearchBot discovers pages for ChatGPT search results. You can block one and allow the other.

 Robots.txt pointing crawlers to a sitemap index while crawl permissions remain independent

How to Add a Sitemap to robots.txt and Verify It

Add an absolute Sitemap URL to robots.txt, then verify that both the robots.txt file and the sitemap resolve on the intended production hostname. The directive helps crawlers discover the sitemap; it does not guarantee crawling or indexing.

 Website allowing an AI search discovery route while blocking a separate model-training route

How to Allow AI Search but Block AI Training

Block only the documented training or product-use controls, such as GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended. Leave AI search crawlers unblocked unless an existing rule blocks them; add explicit search groups only when necessary, and repeat any private-path restrictions inside those groups. Then test the live file by crawler and path.

 Website selectively blocking AI training crawler routes while keeping search discovery routes open

How to Block AI Crawlers with robots.txt

Block only the exact crawler names that match your policy, keep ordinary search and user-requested access separate when you still want them, publish the merged file at the correct host root, and verify each important crawler and path on the live site.

 Diagnostic flow checking a crawler token, URL rule match, server response, and access logs

How to Check Whether AI Crawlers Are Blocked

Fetch the live robots.txt file, select the crawler’s exact product token, and test the specific URL against the matching group and longest Allow or Disallow rule. Then check HTTP responses, caching, firewall rules, and logs, because a crawler may be allowed by robots.txt yet blocked by the server—or disallowed by robots.txt while an unidentified scraper still reaches the page.

 Automatic Perplexity search crawler compared with a single user-triggered page request

PerplexityBot vs Perplexity-User: robots.txt Controls

PerplexityBot is the automatic crawler used to surface and link sites in Perplexity search results. Perplexity-User supports user-requested fetches; Perplexity states that this fetcher generally ignores robots.txt because the request is user initiated.

 Public pages moving through AI search discovery and technical access toward possible source citation

robots.txt for AI Search: Discovery Crawler Guide

If AI-search visibility matters, identify the agents used for search/discovery separately from training crawlers. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Applebot have discovery roles, while similarly named training or user-triggered agents may need different policies.

 Website blocking selected AI training and dataset routes while keeping search discovery open

robots.txt for AI Training: Build a Selective Opt-Out

To restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.

 Publisher website allowing search discovery, blocking selected AI training access, and protecting subscriber content with server controls

robots.txt for Publishers and Content Sites

Publishers should use exact crawler groups instead of a blanket block: keep ordinary search crawling open, decide separately on AI training and AI-search access, and protect subscriber-only material with authentication rather than robots.txt.

 Crawler receiving successful, redirected, missing, and temporary-failure responses for robots.txt

robots.txt HTTP Status Codes: 200, 3xx, 4xx, and 5xx

A successful 2xx response lets crawlers parse robots.txt. Redirects may be followed, ordinary 4xx responses often mean no usable robots.txt restrictions, while 5xx and network failures are treated as temporary retrieval problems. Exact behavior and cache timing vary by crawler, so return a stable 200 response for a valid file and never use 401 or 403 as a substitute for crawl rules.

 URL paths passing through wildcard, exact-ending, and longest-match robots.txt rule selection

robots.txt Wildcards: * and $ Pattern Matching

In robots.txt path rules, * matches zero or more characters and $ anchors the match to the end. The most specific applicable rule decides the result, not the last line in the file. Build the complete policy, then test exact crawler and URL combinations before publishing.

FAQ

Which guide should I read first?

Start with the guide that matches your goal: blocking AI crawlers, allowing AI search while blocking training, or checking whether a live rule works.

Are the comparison guides only about bot names?

No. They explain why providers publish separate tokens and how blocking one purpose can affect training, search visibility, or user-directed retrieval differently.

Do I need technical server access to use the guides?

Not always. Many changes can be made through a CMS or platform, but verification may require checking the live response, headers, logs, or hosting configuration.

Can the guides guarantee that every crawler will comply?

No. They explain robots.txt behavior for compliant crawlers and identify cases where server-side controls are required.

How often should I recheck my robots.txt policy?

Review it after provider documentation changes, platform or plugin updates, domain migrations, major content-policy changes, and unexpected crawler traffic.