To restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: /
A useful policy begins with purpose, not a copied block list. Some agents crawl material that may contribute to model development, some build searchable indexes, and others fetch a page after a user asks a question. Blocking all agents associated with an AI company can unnecessarily remove search citations or user-requested access.
Create a short inventory of the exact published tokens that matter to your content. Examples include GPTBot for OpenAI model development and ClaudeBot for Anthropic model development. Dataset crawlers such as CCBot may also be relevant to a broader data-use policy. Product controls such as Google-Extended and Applebot-Extended require separate interpretation because they are not ordinary standalone crawler identities.
Place each restricted token in its own group so your intent is visible and maintainable:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /A global rule such as User-agent: * Disallow: / requests that every compliant crawler stop, including Googlebot, Bingbot, AI search crawlers, monitoring tools, and other services. That is rarely the intended result for a public site.
If you want public content to remain discoverable in search and AI answers, keep search-oriented agents open in separate groups:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /Provider documentation can change, so do not assume that every bot name containing “AI” is used for training. Review the current purpose before adding a restriction.
Agents such as ChatGPT-User, Claude-User, and Perplexity-User act in response to a person. Their providers may state that robots.txt does not apply or may not reliably apply to these requests. Decide whether user-requested retrieval supports your audience, but do not rely on robots.txt when the content must be technically inaccessible.
The Robots Exclusion Protocol is an advisory standard for automatic clients. It is not authorization, copyright enforcement, or a private-content barrier. A compliant crawler can honor your preference; an unidentified scraper can ignore it. Use login requirements, signed URLs, rate limiting, bot management, or contractual controls where appropriate.
For a ready-made structure, start with the Allow Search, Block AI Training template.
No. It communicates a preference to compliant agents. It does not prevent a crawler from ignoring the file, using previously obtained material, or receiving content through another authorized source.
Usually not. Providers often separate training, search-discovery, and user-requested agents. Blocking every token can reduce citations, search visibility, or features that a person explicitly requested.
They are product-use control tokens rather than ordinary standalone crawler identities. Follow the provider’s current documentation and do not expect a separate matching HTTP user agent in logs.
They support ordinary search discovery. Blocking them can damage search visibility and is broader than an AI-training opt-out.
Use server-side authorization, authentication, paywalls, signed access, rate limits, WAF or bot-management controls. Robots.txt is not a security mechanism.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
AI Training Crawler: Definition and robots.txt ControlAn AI training crawler collects public web content that may contribute to model development, improvement, evaluation, or safety work. A crawler-specific Disallow rule can signal that future collection is not permitted, but it does not erase previously acquired data or secure private content.
GPTBot robots.txt GuideGPTBot is OpenAI’s automated crawler for content that may be used to improve and train generative AI foundation models. To opt a site out of future GPTBot crawling, add a specific GPTBot group to robots.txt. This setting is separate from OAI-SearchBot and ChatGPT-User.
ClaudeBot robots.txt GuideClaudeBot is Anthropic’s crawler for public web content that may contribute to model development and training. A ClaudeBot-specific Disallow rule signals that future material should be excluded from Anthropic training crawls. Claude-SearchBot and Claude-User are separate agents with different purposes.