Allow Search, Block AI Training robots.txt Template
Use separate user-agent groups: block documented training or dataset agents such as GPTBot and ClaudeBot, while allowing ordinary search and dedicated AI-search crawlers.
Copy-paste robots.txt example
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Applebot Allow: / User-agent: * Allow: /
Copy-paste template
This template keeps ordinary search engines and several documented AI-search crawlers open while asking named training or data-collection agents not to crawl the site. Copy it to the root-level /robots.txt file, then adjust the list to match your own policy.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: Applebot
Allow: /
User-agent: *
Allow: /What the blocked groups control
GPTBot is OpenAI's training-oriented crawler, while ClaudeBot is Anthropic's model-development crawler. Google-Extended and Applebot-Extended are product-control tokens that let publishers restrict specified model-training or generative-AI uses without blocking ordinary Googlebot or Applebot search crawling.
CCBot collects content for the open Common Crawl repository. That dataset can support many downstream uses, including research, search, and model development, so blocking it is a broader decision than blocking a provider-specific training agent.
What remains open
The example explicitly allows Googlebot, Bingbot, and Applebot for conventional discovery. It also allows OAI-SearchBot, Claude-SearchBot, and PerplexityBot, which their operators document for search discovery or answer visibility rather than foundation-model training.
Allowing a search crawler does not guarantee ranking, citation, referral traffic, or inclusion in an AI answer. It only permits the compliant crawler to request allowed URLs.
User-requested fetchers need separate consideration
Some providers use agents that fetch a page because a person asked an assistant to visit it. Examples include ChatGPT-User, Claude-User, and Perplexity-User. Their operators describe different robots.txt behavior, and some user-initiated fetches may not follow the same rules as automatic crawling. Do not treat robots.txt as enforceable access control for private or licensed material.
Install the template safely
- Place the file at
https://example.com/robots.txt, not inside a subfolder. - Keep one separate group for each documented token.
- Replace the sample domain only when adding a
Sitemap:line; user-agent rules themselves do not contain a domain. - Test representative public and restricted URLs with the robots.txt checker.
- Repeat the policy on every subdomain or host that serves content.
Review the list instead of treating it as permanent
Crawler names and product purposes change. Recheck official provider documentation periodically and update the file when you add a new content host or change your search and training policy. The selective AI access guide explains the decision in more depth, while the generator can create a configuration tailored to your choices.
FAQ
Can I allow ChatGPT search while blocking OpenAI training?
Yes. OpenAI documents independent controls: allow OAI-SearchBot for ChatGPT search visibility and disallow GPTBot to express a training opt-out.
Will Google Search remain open if I block Google-Extended?
Yes. Google states that Google-Extended is a separate control token and does not affect inclusion or ranking in Google Search when Googlebot remains allowed.
Why is CCBot included in the blocked section?
CCBot contributes to the open Common Crawl repository, which can be reused for several downstream purposes. Blocking it is broader than a provider-specific training opt-out and may reduce some downstream discovery.
Does this template protect paid or private content?
No. Robots.txt is a public instruction for compliant crawlers. Use authentication, authorization, paywall logic, and server-side controls for content that must not be retrieved.
Do I need the final User-agent: * group?
It is not required when the default policy is open, but it makes the intended fallback clear. Review any existing wildcard restrictions before replacing your current file.
Related tools
Related pages
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
robots.txt for AI Training: Build a Selective Opt-OutTo restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.
robots.txt for AI Search Visibility and CitationsTo support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
AI Robots.txt GeneratorChoose a policy mode, enter your website and sitemap, add any path or crawler overrides, then generate and download a robots.txt file. Publish it at the root of the correct host and verify the live rules with the checker before relying on them.