ClaudeBot robots.txt Rules for Anthropic Training Crawling

ClaudeBot’s role in Anthropic’s crawler system

Anthropic operates three documented robots with different jobs. ClaudeBot collects public web content that could contribute to model training and improvement. Claude-SearchBot supports search-result quality, while Claude-User retrieves pages in response to a person’s request.

Because the tokens are separate, a site owner can write a policy that blocks training crawling without automatically disabling Claude search discovery or user-directed retrieval. The example on this page shows that targeted approach.

Block ClaudeBot without overblocking

To opt an entire host out of ClaudeBot crawling, place a Disallow: / rule under User-agent: ClaudeBot. Apply the policy independently to every subdomain that needs the same restriction because each host can publish its own robots.txt file.

Avoid using only User-agent: * when your intention concerns Anthropic training crawling alone. A wildcard policy can affect ordinary search engines, monitoring tools, and other compliant bots. Keep Claude-SearchBot and Claude-User in separate groups when their access decision differs.

Anthropic also supports crawl pacing

Anthropic states that its bots honor standard robots.txt directives and supports the non-standard Crawl-delay extension. If server load rather than training permission is the problem, a measured crawl delay may be more appropriate than a complete site-wide block. Support for Crawl-delay is crawler-specific, so do not assume every search engine interprets it the same way.

A crawl-delay instruction and a disallow rule solve different problems: one limits request pace, while the other tells a compliant bot not to fetch matching paths.

Do not rely on IP blocking as the opt-out method

Anthropic advises site owners to express the opt-out in robots.txt. Blocking network addresses alone can prevent the bot from reading the policy and may not provide a persistent opt-out as address ranges change. IP information is still useful for authenticating log entries, but robots.txt should carry the crawler preference.

Recommended verification process

1. Merge the ClaudeBot rule with the site’s existing robots.txt file.
2. Decide separately whether Claude-SearchBot and Claude-User should be allowed.
3. Publish the file at the root of each relevant host.
4. Confirm that the URL returns the intended plain-text content and a successful response.
5. Test the final rule with the robots.txt checker and review server logs after deployment.

The ClaudeBot vs Claude-SearchBot guide covers the policy split, and the Claude crawler template provides a broader starting point.

Anthropic’s current definitions, examples, and crawl-delay guidance are available in its official crawler guidance.