ClaudeBot and Claude-SearchBot are separate Anthropic bots. ClaudeBot collects public web content that may contribute to model training, while Claude-SearchBot indexes content to improve Claude search results. You can disallow ClaudeBot while allowing Claude-SearchBot.
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
Anthropic documents three web-access roles: model development, search optimization, and user-directed retrieval. ClaudeBot is associated with public-web collection that could contribute to training. Claude-SearchBot navigates the web to improve the relevance and accuracy of search results. Claude-User supports requests initiated by Claude users.
For robots.txt policy, the key point is that the first two bots are independent. Blocking ClaudeBot does not require blocking Claude-SearchBot, and allowing Claude-SearchBot does not grant ClaudeBot access.
ClaudeBot.Disallow: /.The rule applies to compliant future crawling on the exact host. It does not delete prior copies, protect private data, or block clients that misidentify themselves.
Claude-SearchBot.Publish separate groups when you want Claude search discovery but do not want ClaudeBot training access:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /Add rules for Claude-User only after making a separate decision about user-requested access. Anthropic says disabling Claude-User prevents retrieval in response to a user query and may reduce visibility in user-directed web search.
Anthropic states that its bots honor standard robots.txt directives and support the non-standard Crawl-delay extension where appropriate. Use a modest delay only when logs show that crawl rate is a real server problem. A very high value can slow useful discovery without solving underlying performance issues.
Publish the intended robots.txt file on every subdomain you want to control. A policy at www.example.com/robots.txt does not automatically apply to docs.example.com or another protocol and port.
See Allow AI Search but Block AI Training for a multi-provider policy and Claude-User robots.txt and User-Requested Access for the third Anthropic agent.
No. Anthropic describes ClaudeBot as a model-development crawler and Claude-SearchBot as a crawler for improving search-result relevance and accuracy.
Yes. Put the two tokens in separate robots.txt groups, disallow ClaudeBot, and allow Claude-SearchBot.
Anthropic says this prevents indexing for search optimization and may reduce the site’s visibility and accuracy in Claude user search results.
Yes. Anthropic states that it supports the non-standard Crawl-delay extension where appropriate. Use it only when measured crawl load requires throttling.
No. Claude-User is a separate agent for user-directed retrieval. Decide and configure its access independently.
ClaudeBot robots.txt GuideClaudeBot is Anthropic’s crawler for public web content that may contribute to model development and training. A ClaudeBot-specific Disallow rule signals that future material should be excluded from Anthropic training crawls. Claude-SearchBot and Claude-User are separate agents with different purposes.
Claude-SearchBot robots.txt RulesClaude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.
Claude-User robots.txt and User-Requested AccessClaude-User is used when a person asks Claude to retrieve web content. Anthropic says it honors robots.txt, but robots.txt is only a crawler preference—not security. Protect private pages with authentication and authorization.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
Block Claude Crawlers robots.txt TemplateTo block Anthropic’s documented Claude agents, publish separate Disallow groups for ClaudeBot, Claude-SearchBot, and Claude-User. Blocking all three also removes training, Claude search discovery, and user-directed retrieval, so use only the groups that match your policy.