ClaudeBot vs Claude-SearchBot: Training vs Search

ClaudeBot and Claude-SearchBot are separate Anthropic bots. ClaudeBot collects public web content that may contribute to model training, while Claude-SearchBot indexes content to improve Claude search results. You can disallow ClaudeBot while allowing Claude-SearchBot.

Independent technical reference Updated Editorial approach Report a correction
Separate ClaudeBot training, Claude-SearchBot discovery, and secondary user-request access routes

Copy-paste robots.txt example

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Why Anthropic uses separate bot identities

Anthropic documents three web-access roles: model development, search optimization, and user-directed retrieval. ClaudeBot is associated with public-web collection that could contribute to training. Claude-SearchBot navigates the web to improve the relevance and accuracy of search results. Claude-User supports requests initiated by Claude users.

For robots.txt policy, the key point is that the first two bots are independent. Blocking ClaudeBot does not require blocking Claude-SearchBot, and allowing Claude-SearchBot does not grant ClaudeBot access.

ClaudeBot

  • Purpose: collect web content that may contribute to model development and training.
  • User-agent token: ClaudeBot.
  • Full-site restriction: a ClaudeBot group with Disallow: /.
  • Meaning of opt-out: Anthropic states that restricting ClaudeBot signals that future site materials should be excluded from its AI model-training datasets.

The rule applies to compliant future crawling on the exact host. It does not delete prior copies, protect private data, or block clients that misidentify themselves.

Claude-SearchBot

  • Purpose: index and analyze public content to improve Claude search-result quality.
  • User-agent token: Claude-SearchBot.
  • Visibility effect: Anthropic says disabling it may reduce a site’s visibility and the accuracy of its representation in user search results.
  • Guarantee: allowing access does not guarantee that the site will be selected, cited, or ranked for a query.

Common selective policy

Publish separate groups when you want Claude search discovery but do not want ClaudeBot training access:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

Add rules for Claude-User only after making a separate decision about user-requested access. Anthropic says disabling Claude-User prevents retrieval in response to a user query and may reduce visibility in user-directed web search.

Crawl-delay and subdomains

Anthropic states that its bots honor standard robots.txt directives and support the non-standard Crawl-delay extension where appropriate. Use a modest delay only when logs show that crawl rate is a real server problem. A very high value can slow useful discovery without solving underlying performance issues.

Publish the intended robots.txt file on every subdomain you want to control. A policy at www.example.com/robots.txt does not automatically apply to docs.example.com or another protocol and port.

How to test the policy

  1. Open the public robots.txt file and confirm the exact token spelling.
  2. Test a public URL as Claude-SearchBot and confirm it is allowed.
  3. Test the same URL as ClaudeBot and confirm it is blocked.
  4. Use the checker to detect group and path mistakes.
  5. Review current Anthropic documentation before treating an old block list as permanent.

See Allow AI Search but Block AI Training for a multi-provider policy and Claude-User robots.txt and User-Requested Access for the third Anthropic agent.

FAQ

Are ClaudeBot and Claude-SearchBot the same bot?

No. Anthropic describes ClaudeBot as a model-development crawler and Claude-SearchBot as a crawler for improving search-result relevance and accuracy.

Can I block ClaudeBot but allow Claude-SearchBot?

Yes. Put the two tokens in separate robots.txt groups, disallow ClaudeBot, and allow Claude-SearchBot.

What happens if I block Claude-SearchBot?

Anthropic says this prevents indexing for search optimization and may reduce the site’s visibility and accuracy in Claude user search results.

Does Anthropic support Crawl-delay?

Yes. Anthropic states that it supports the non-standard Crawl-delay extension where appropriate. Use it only when measured crawl load requires throttling.

Is Claude-User covered by the Claude-SearchBot rule?

No. Claude-User is a separate agent for user-directed retrieval. Decide and configure its access independently.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Website with separate routes for blocked training crawling, open search discovery, and user-requested access ClaudeBot robots.txt Guide

ClaudeBot is Anthropic’s crawler for public web content that may contribute to model development and training. A ClaudeBot-specific Disallow rule signals that future material should be excluded from Anthropic training crawls. Claude-SearchBot and Claude-User are separate agents with different purposes.

 Public website pages available to Claude search while a separate AI training route is blocked Claude-SearchBot robots.txt Rules

Claude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.

 Single user-requested AI fetch reaching a public webpage while private content remains behind server-side access control Claude-User robots.txt and User-Requested Access

Claude-User is used when a person asks Claude to retrieve web content. Anthropic says it honors robots.txt, but robots.txt is only a crawler preference—not security. Protect private pages with authentication and authorization.

 Website allowing an AI search discovery route while blocking a separate model-training route How to Allow AI Search but Block AI Training

Block only the documented training or product-use controls, such as GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended. Leave AI search crawlers unblocked unless an existing rule blocks them; add explicit search groups only when necessary, and repeat any private-path restrictions inside those groups. Then test the live file by crawler and path.

 Three separate Claude crawler routes blocked by independent robots.txt rules while regular search remains open Block Claude Crawlers robots.txt Template

To block Anthropic’s documented Claude agents, publish separate Disallow groups for ClaudeBot, Claude-SearchBot, and Claude-User. Blocking all three also removes training, Claude search discovery, and user-directed retrieval, so use only the groups that match your policy.