To block Anthropic’s documented Claude agents, publish separate Disallow groups for ClaudeBot, Claude-SearchBot, and Claude-User. Blocking all three also removes training, Claude search discovery, and user-directed retrieval, so use only the groups that match your policy.
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: /
Anthropic documents three separate agents because they serve different purposes. ClaudeBot collects public web content that may contribute to model development, Claude-SearchBot supports search discovery, and Claude-User retrieves pages in response to a person’s request. If you want to block all three categories, use an explicit group for every token:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /Publish the rules at the root of the relevant host, such as https://example.com/robots.txt. Do not place the file inside a theme, content folder, or administration directory.
A complete block is not always the best publishing policy. Each group can be used independently:
Do not add a broad User-agent: * block unless you intend to restrict ordinary search crawlers and every other compliant bot that relies on the wildcard group.
The Claude groups above do not block Googlebot or Bingbot. You normally do not need extra Allow: / groups for those search engines unless your existing robots.txt contains a wildcard restriction that would otherwise apply. Review the complete file before publishing because a broader existing rule can change the final result for unrelated crawlers.
/robots.txt URL and confirm that the response is plain text with HTTP 200.Robots.txt is a policy instruction for compliant automated clients, not authentication or a firewall. If a page must be inaccessible regardless of the client’s identity, enforce that requirement with login, authorization, server rules, or another real access-control mechanism.
No. ClaudeBot, Claude-SearchBot, and Claude-User have different purposes. Include only the groups that match your training, search visibility, and user-requested access policy.
No. Anthropic documents ClaudeBot and Claude-SearchBot separately. Blocking ClaudeBot alone does not express a block for Claude-SearchBot or Claude-User.
Not by itself. A crawler-specific Claude-SearchBot group does not apply to Googlebot. Avoid a broad wildcard block if you want conventional search to remain available.
The new rule is applied after Anthropic refreshes the file. Verify the live response immediately, then allow time for crawler behavior and logs to reflect the change.
No. Robots.txt is a voluntary crawling instruction. Protect private, paid, or sensitive resources with authentication, authorization, or server-side access controls.
ClaudeBot robots.txt GuideClaudeBot is Anthropic’s crawler for public web content that may contribute to model development and training. A ClaudeBot-specific Disallow rule signals that future material should be excluded from Anthropic training crawls. Claude-SearchBot and Claude-User are separate agents with different purposes.
Claude-SearchBot robots.txt RulesClaude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.
Claude-User robots.txt and User-Requested AccessClaude-User is used when a person asks Claude to retrieve web content. Anthropic says it honors robots.txt, but robots.txt is only a crawler preference—not security. Protect private pages with authentication and authorization.
ClaudeBot vs Claude-SearchBot: Training vs SearchClaudeBot and Claude-SearchBot are separate Anthropic bots. ClaudeBot collects public web content that may contribute to model training, while Claude-SearchBot indexes content to improve Claude search results. You can disallow ClaudeBot while allowing Claude-SearchBot.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.