ClaudeBot vs Claude-SearchBot vs Claude-User

Anthropic uses three distinct agents: ClaudeBot for model-development crawling, Claude-SearchBot for search quality and indexing, and Claude-User for user-directed retrieval. Treat them as separate policy decisions.

Independent technical reference Updated Editorial approach Report a correction
Separate ClaudeBot training, Claude-SearchBot discovery, and secondary user-request access routes

Copy-paste robots.txt example

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Anthropic publishes separate user agents because “allow Claude” is not a single web-access decision. The three agents represent different purposes, and a robots.txt file can express a different choice for each one.

Claude policy profiles with measurable results

Desired outcomeClaudeBotClaude-SearchBotClaude-User
Opt out of automated training crawl; keep searchDisallowAllowDecide separately
Limit automatic search discovery; permit training collectionAllowDisallowDecide separately
Block both automatic purposesDisallowDisallowDecide separately

Each cell is a distinct policy decision, not an implied inheritance rule. Write a group for the agent you want to control, then test / and an inner content URL for all three tokens in the draft tester. Check the winning rule rather than only the final allowed/blocked badge.

Anthropic describes Claude-User as supporting user-initiated access, which is not the same task as automatic search indexing or model-training collection. Its crawler reference is the authority for the current distinction. None of these robots.txt settings replaces authentication, server-side authorization or protection of paid/private URLs.

The three Anthropic agents

AgentPrimary purposeWhat a block changes
ClaudeBotCollect public web content that could contribute to model training.Signals that future site material should be excluded from Anthropic’s training datasets.
Claude-SearchBotNavigate and analyze the web to improve Claude search result quality.Can reduce the site’s indexing/visibility in Claude search experiences.
Claude-UserRetrieve web content at a Claude user’s direction.Prevents Anthropic from retrieving the site for those user-directed requests when the rule is honored.

Example: block training, keep search and user retrieval

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

This is useful when the business goal is discovery in Claude experiences without allowing the training crawler.

Why this is different from a ClaudeBot reference page

The ClaudeBot reference answers “what is this crawler?” This guide answers a policy question: which Anthropic agent should I allow or block for a specific outcome?

Operational checks

Anthropic states that its bots respect robots.txt and provides published bot information for site owners. After changing a policy, verify the exact group names in the live file and test representative URLs with the robots.txt Checker. Use server/WAF controls when you need stronger enforcement than cooperative crawler rules provide.

Selective path rules are possible

You can also allow an Anthropic agent generally while disallowing a specific path, or do the reverse where supported by normal robots matching. Keep the purpose of each agent visible in the file so future maintenance does not collapse three different choices into one blanket Claude rule.

FAQ

Are ClaudeBot and Claude-SearchBot the same bot?

No. Anthropic describes ClaudeBot as a model-development crawler and Claude-SearchBot as a crawler for improving search-result relevance and accuracy.

Can I block ClaudeBot but allow Claude-SearchBot?

Yes. Put the two tokens in separate robots.txt groups, disallow ClaudeBot, and allow Claude-SearchBot.

What happens if I block Claude-SearchBot?

Anthropic says this prevents indexing for search optimization and may reduce the site’s visibility and accuracy in Claude user search results.

Does Anthropic support Crawl-delay?

Yes. Anthropic states that it supports the non-standard Crawl-delay extension where appropriate. Use it only when measured crawl load requires throttling.

Is Claude-User covered by the Claude-SearchBot rule?

No. Claude-User is a separate agent for user-directed retrieval. Decide and configure its access independently.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Website with separate routes for blocked training crawling, open search discovery, and user-requested access ClaudeBot robots.txt Guide

ClaudeBot is Anthropic’s crawler for public web content that may contribute to model development and training. A ClaudeBot-specific Disallow rule signals that future material should be excluded from Anthropic training crawls. Claude-SearchBot and Claude-User are separate agents with different purposes.

 Public website pages available to Claude search while a separate AI training route is blocked Claude-SearchBot robots.txt Rules

Claude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.

 Single user-requested AI fetch reaching a public webpage while private content remains behind server-side access control Claude-User robots.txt and User-Requested Access

Claude-User is used when a person asks Claude to retrieve web content. Anthropic says it honors robots.txt, but robots.txt is only a crawler preference—not security. Protect private pages with authentication and authorization.

 Website allowing an AI search discovery route while blocking a separate model-training route How to Allow AI Search but Block AI Training

Block only the documented training or product-use controls, such as GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended. Leave AI search crawlers unblocked unless an existing rule blocks them; add explicit search groups only when necessary, and repeat any private-path restrictions inside those groups. Then test the live file by crawler and path.

 Three separate Claude crawler routes blocked by independent robots.txt rules while regular search remains open Block Claude Crawlers robots.txt Template

To block Anthropic’s documented Claude agents, publish separate Disallow groups for ClaudeBot, Claude-SearchBot, and Claude-User. Blocking all three also removes training, Claude search discovery, and user-directed retrieval, so use only the groups that match your policy.