Claude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
Claude-SearchBot is Anthropic’s crawler for discovering and analyzing public web content used to improve the relevance and accuracy of Claude search responses. It is not the same agent as ClaudeBot, which is associated with model-development crawling, or Claude-User, which retrieves a page after an individual user asks Claude to access it.
This separation lets a publisher make a precise choice: keep Claude search discovery available while opting out of future training crawls. Anthropic states that its bots honor standard robots.txt directives and that disabling Claude-SearchBot may reduce a site’s visibility and accuracy in user search results.
For many publishers, the useful configuration is to block ClaudeBot but allow Claude-SearchBot. The example on this page does exactly that and keeps Claude-User available for user-directed retrieval. You can generate a broader policy with the robots.txt generator and test the published file with the checker.
If you want Claude search excluded completely, replace the Claude-SearchBot Allow: / line with Disallow: /. To exclude only sensitive sections, list those paths under the Claude-SearchBot group instead of blocking the whole host.
Blocking Claude-SearchBot concerns Anthropic’s search discovery. It does not block Googlebot, Bingbot, or ordinary search indexing unless your robots.txt file also contains rules for those agents. It also does not protect confidential content: authentication, authorization, and server-side access controls are still required for private pages.
Anthropic documents support for the non-standard Crawl-delay directive where appropriate. Use it only when request frequency is a real server concern. Robots.txt applies per host, so a rule on example.com does not automatically cover docs.example.com or another subdomain.
https://your-domain.com/robots.txt.User-agent: * rule accidentally blocks public content.For a direct comparison of the Anthropic agents, see ClaudeBot vs Claude-SearchBot.
Anthropic uses Claude-SearchBot to discover and analyze public web content for better search relevance and accuracy in Claude search experiences.
Yes. They use separate robots.txt tokens, so you can allow Claude-SearchBot for search discovery and disallow ClaudeBot for model-development crawling.
No, not by itself. Google Search uses Googlebot. Google visibility changes only if your robots.txt rules also restrict Googlebot or use an overly broad group.
Anthropic documents support for the non-standard Crawl-delay directive where appropriate, although it should be used only when crawl frequency is a genuine concern.
Yes. Robots.txt is scoped to a specific host, protocol, and port, so each subdomain that needs a policy should expose its own robots.txt file.
ClaudeBot robots.txt GuideClaudeBot is Anthropic’s crawler for public web content that may contribute to model development and training. A ClaudeBot-specific Disallow rule signals that future material should be excluded from Anthropic training crawls. Claude-SearchBot and Claude-User are separate agents with different purposes.
Claude-User robots.txt and User-Requested AccessClaude-User is used when a person asks Claude to retrieve web content. Anthropic says it honors robots.txt, but robots.txt is only a crawler preference—not security. Protect private pages with authentication and authorization.
Block Claude Crawlers robots.txt TemplateTo block Anthropic’s documented Claude agents, publish separate Disallow groups for ClaudeBot, Claude-SearchBot, and Claude-User. Blocking all three also removes training, Claude search discovery, and user-directed retrieval, so use only the groups that match your policy.
ClaudeBot vs Claude-SearchBot: Training vs SearchClaudeBot and Claude-SearchBot are separate Anthropic bots. ClaudeBot collects public web content that may contribute to model training, while Claude-SearchBot indexes content to improve Claude search results. You can disallow ClaudeBot while allowing Claude-SearchBot.
robots.txt for AI Search Visibility and CitationsTo support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.