Claude-User robots.txt and User-Requested Web Access

Claude-User is used when a person asks Claude to retrieve web content. Anthropic says it honors robots.txt, but robots.txt is only a crawler preference—not security. Protect private pages with authentication and authorization.

Independent technical reference Updated Editorial approach Report a correction
Single user-requested AI fetch reaching a public webpage while private content remains behind server-side access control

Primary reference

Official crawler documentation

Use the provider's current documentation to confirm the crawler token and purpose before changing a production robots.txt file.

Open Anthropic crawler documentation Reference checked July 6, 2026

Copy-paste robots.txt example

User-agent: Claude-User
Disallow: /account/
Disallow: /checkout/
Disallow: /members/

What Claude-User represents

Claude-User identifies web requests made in response to an individual using Claude. This is different from Claude-SearchBot’s automated search discovery and ClaudeBot’s model-development crawling. A Claude-User request is tied to a user action, such as asking Claude to read, summarize, or reference a public page.

Anthropic provides a separate user-agent token so site owners can decide whether these user-directed requests may access their content. Blocking Claude-User can prevent Claude from retrieving pages in response to a user query and may reduce the site’s visibility in user-directed web search.

What robots.txt can control

You can disallow the whole site or list selected paths under User-agent: Claude-User. The example below leaves ordinary public pages available while identifying account, checkout, and member areas as off-limits. Adjust the paths to match your real site structure.

Because robots.txt is public and advisory, it must never be the only protection for private content. Login checks, authorization rules, signed URLs, paywalls, and server-side restrictions must enforce access whether the request comes from Claude-User, a browser, or an unknown scraper.

Claude-User versus the other Anthropic agents

  • ClaudeBot: automated collection that may contribute to model development and training.
  • Claude-SearchBot: automated discovery for better Claude search results.
  • Claude-User: a fetch performed because a person asked Claude to access a page.

These tokens work independently. A publisher can block ClaudeBot, allow Claude-SearchBot, and restrict Claude-User only on sensitive paths. The Claude crawler template is a starting point, while the generator is better for a custom mixed policy.

Practical setup and verification

  1. List every genuinely private path and protect it at the application or server level.
  2. Add matching Claude-User directives to communicate your crawler preference.
  3. Publish robots.txt at the root of each relevant host or subdomain.
  4. Run the checker and inspect the live file rather than trusting a local draft.

If you need the broader distinction between crawler instructions and hard access control, read Does robots.txt Stop AI Scraping?.

FAQ

Is Claude-User an automated training crawler?

No. Claude-User is associated with a specific user-directed request. ClaudeBot is Anthropic’s separate agent for collection that may contribute to model development and training.

Does Claude-User obey robots.txt?

Anthropic states that its bots honor standard robots.txt directives, including the separate Claude-User token.

Can robots.txt protect a private account page?

No. It can communicate a preference to compliant agents, but private pages need authentication, authorization, and server-side enforcement.

What happens if I block Claude-User?

Claude may be unable to retrieve your content when a user asks for it, which can reduce visibility in user-directed web search and answers.

Can I block only selected paths for Claude-User?

Yes. Add specific Disallow paths under the Claude-User group and keep public content available by default.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Website with separate routes for blocked training crawling, open search discovery, and user-requested access ClaudeBot robots.txt Guide

ClaudeBot is Anthropic’s crawler for public web content that may contribute to model development and training. A ClaudeBot-specific Disallow rule signals that future material should be excluded from Anthropic training crawls. Claude-SearchBot and Claude-User are separate agents with different purposes.

 Public website pages available to Claude search while a separate AI training route is blocked Claude-SearchBot robots.txt Rules

Claude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.

 Three separate Claude crawler routes blocked by independent robots.txt rules while regular search remains open Block Claude Crawlers robots.txt Template

To block Anthropic’s documented Claude agents, publish separate Disallow groups for ClaudeBot, Claude-SearchBot, and Claude-User. Blocking all three also removes training, Claude search discovery, and user-directed retrieval, so use only the groups that match your policy.

 Separate ClaudeBot training, Claude-SearchBot discovery, and secondary user-request access routes ClaudeBot vs Claude-SearchBot: Training vs Search

ClaudeBot and Claude-SearchBot are separate Anthropic bots. ClaudeBot collects public web content that may contribute to model training, while Claude-SearchBot indexes content to improve Claude search results. You can disallow ClaudeBot while allowing Claude-SearchBot.

 Voluntary robots.txt crawler rules shown separately from enforceable server-side access controls Does robots.txt Stop AI Scraping? Limits and Better Controls

Robots.txt can stop compliant AI crawlers from requesting selected URLs, but it cannot enforce access against unidentified scrapers, spoofed user agents, browsers, or tools that ignore the protocol. Use precise crawler rules for policy, then add authentication, authorization, rate limits, or firewall controls where access must actually be prevented.