Claude-SearchBot robots.txt Rules for Claude Search

Claude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.

Independent technical reference Updated July 15, 2026 Editorial approach Report a correction

Primary reference

Official crawler documentation

Use the provider's current documentation to confirm the crawler token and purpose before changing a production robots.txt file.

Open Anthropic crawler documentation Reference checked July 6, 2026

Copy-paste robots.txt example

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

What Claude-SearchBot does

Claude-SearchBot is Anthropic’s crawler for discovering and analyzing public web content used to improve the relevance and accuracy of Claude search responses. It is not the same agent as ClaudeBot, which is associated with model-development crawling, or Claude-User, which retrieves a page after an individual user asks Claude to access it.

This separation lets a publisher make a precise choice: keep Claude search discovery available while opting out of future training crawls. Anthropic states that its bots honor standard robots.txt directives and that disabling Claude-SearchBot may reduce a site’s visibility and accuracy in user search results.

Recommended selective policy

For many publishers, the useful configuration is to block ClaudeBot but allow Claude-SearchBot. The example on this page does exactly that and keeps Claude-User available for user-directed retrieval. You can generate a broader policy with the robots.txt generator and test the published file with the checker.

If you want Claude search excluded completely, replace the Claude-SearchBot Allow: / line with Disallow: /. To exclude only sensitive sections, list those paths under the Claude-SearchBot group instead of blocking the whole host.

What blocking changes—and what it does not

Blocking Claude-SearchBot concerns Anthropic’s search discovery. It does not block Googlebot, Bingbot, or ordinary search indexing unless your robots.txt file also contains rules for those agents. It also does not protect confidential content: authentication, authorization, and server-side access controls are still required for private pages.

Crawl rate and subdomains

Anthropic documents support for the non-standard Crawl-delay directive where appropriate. Use it only when request frequency is a real server concern. Robots.txt applies per host, so a rule on example.com does not automatically cover docs.example.com or another subdomain.

How to verify the result

Publish the file at https://your-domain.com/robots.txt.
Confirm that the Claude-SearchBot group contains the intended paths.
Check that no broad User-agent: * rule accidentally blocks public content.
Run the site’s robots.txt checker after deployment.

For a direct comparison of the Anthropic agents, see ClaudeBot vs Claude-SearchBot.

FAQ

What is Claude-SearchBot used for?

Anthropic uses Claude-SearchBot to discover and analyze public web content for better search relevance and accuracy in Claude search experiences.

Can I allow Claude-SearchBot while blocking ClaudeBot?

Yes. They use separate robots.txt tokens, so you can allow Claude-SearchBot for search discovery and disallow ClaudeBot for model-development crawling.

Will blocking Claude-SearchBot affect Google Search?

No, not by itself. Google Search uses Googlebot. Google visibility changes only if your robots.txt rules also restrict Googlebot or use an overly broad group.

Does Claude-SearchBot support Crawl-delay?

Anthropic documents support for the non-standard Crawl-delay directive where appropriate, although it should be used only when crawl frequency is a genuine concern.

Do I need separate rules for subdomains?

Yes. Robots.txt is scoped to a specific host, protocol, and port, so each subdomain that needs a policy should expose its own robots.txt file.