Use separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / Sitemap: https://example.com/sitemap.xml
Many publishers want their pages to remain discoverable in AI search while declining future use by model-training crawlers. That is possible only when a provider exposes separate crawler identities for those purposes. A company name by itself is not enough; the robots.txt file must address the published User-agent tokens.
OpenAI documents GPTBot for content that may contribute to training its generative AI foundation models and OAI-SearchBot for surfacing sites in ChatGPT search. Anthropic documents ClaudeBot for model development and Claude-SearchBot for search-result quality. Each token needs its own group.
The following policy asks OpenAI not to use future GPTBot crawling for training while allowing OAI-SearchBot to discover public pages for ChatGPT search:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /OpenAI states that these settings are independent. A robots.txt change may take roughly 24 hours to be reflected by its search systems. Allowing OAI-SearchBot permits discovery; it does not guarantee that a page will be cited or shown for a particular query.
Anthropic also exposes separate training and search agents:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /Blocking ClaudeBot signals that future site materials should be excluded from Anthropic’s model-training datasets. Blocking Claude-SearchBot is a different choice and may reduce the site’s visibility or accuracy in Claude search results.
ChatGPT-User and Claude-User are associated with requests initiated by an individual user. They are not the same as automatic search indexing or training crawling. OpenAI notes that robots.txt may not apply to ChatGPT-User in the same way because the action is user initiated; Anthropic provides Claude-User as a separate controllable agent. Review these tokens as their own policy decision rather than silently grouping them with training bots.
Do not add User-agent: * with Disallow: / to implement this policy. That would also request that Googlebot, Bingbot, and other compliant crawlers stop fetching the site. If conventional search traffic matters, leave those agents open and use narrow groups only for the AI purposes you want to restrict.
Google-Extended is another separate control: it is a product token used to manage certain Gemini-related uses of content and is not a replacement for Googlebot. Review Google-Extended vs Googlebot before changing Google-related rules.
For a reusable starting point, open the Allow Search, Block AI Training template. Treat it as a policy template, not as a permanent substitute for reviewing current provider documentation.
Yes. OpenAI documents GPTBot and OAI-SearchBot as independent settings. Allow OAI-SearchBot and disallow GPTBot to express that policy.
Yes. Use separate groups that disallow ClaudeBot and allow Claude-SearchBot. Anthropic describes these bots as serving different purposes.
No. It permits discovery and indexing for the provider’s search features, but relevance, selection, citation and ranking remain the provider’s decisions.
Usually not. A wildcard full-site block can also affect conventional search crawlers and other useful services. Use exact crawler tokens for a selective policy.
No. Content Signals express preferences about post-access uses, while Allow and Disallow groups control access for crawlers that recognize them. Adoption differs, so the mechanisms can be complementary.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
GPTBot vs OAI-SearchBot: Training vs SearchGPTBot and OAI-SearchBot are separate OpenAI crawler controls. GPTBot crawls content that may be used to improve and train OpenAI’s generative AI foundation models; OAI-SearchBot discovers pages for ChatGPT search results. You can block one and allow the other.
ClaudeBot vs Claude-SearchBot: Training vs SearchClaudeBot and Claude-SearchBot are separate Anthropic bots. ClaudeBot collects public web content that may contribute to model training, while Claude-SearchBot indexes content to improve Claude search results. You can disallow ClaudeBot while allowing Claude-SearchBot.
Allow Search, Block AI Training robots.txt TemplateUse separate user-agent groups: block documented training or dataset agents such as GPTBot and ClaudeBot, while allowing ordinary search and dedicated AI-search crawlers.
Content-Signal in robots.txt: Meaning and ExamplesContent-Signal is a non-standard robots.txt directive for expressing how accessed content may be used, including search indexing, real-time AI input, model training, and newer reuse levels. It states a preference or rights reservation; it does not itself block requests or guarantee that every crawler will comply.