Publishers should use exact crawler groups instead of a blanket block: keep ordinary search crawling open, decide separately on AI training and AI-search access, and protect subscriber-only material with authentication rather than robots.txt.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: /
Publishers, newsrooms, magazines, blogs, research libraries, and other content-heavy sites usually have more than one goal. They may want articles discovered in Google or Bing, cited in AI search, excluded from model-training collection, and protected behind a subscription at the same time. Those goals require separate controls.
Robots.txt is useful for communicating crawler policy to compliant automated clients. It is not a licence agreement, paywall, authentication layer, or guarantee that every scraper will comply. Treat it as one part of a wider publishing and access strategy.
Do not assume that every bot from one provider has the same purpose. OpenAI documents separate controls for GPTBot and OAI-SearchBot. Anthropic similarly distinguishes ClaudeBot, Claude-SearchBot, and user-requested access. Google uses Googlebot for Search and a separate Google-Extended product token for specified Gemini uses.
This means a publisher can make a selective decision: allow ordinary search indexing and AI-search discovery while blocking named training-oriented crawlers. Review each official token rather than blocking a company name or every crawler with User-agent: *.
The following example keeps common search and AI-search routes open while requesting that several training-oriented crawlers stay out:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /Adapt the list to your editorial policy. A publisher that does not want AI-search citations can also block those search-specific agents. A publisher that values referral traffic and attribution may choose to keep them open.
Subscriber articles, licensed archives, embargoed material, account pages, and internal editorial systems need enforceable access controls. Use authentication, authorization, paywall logic, signed URLs, and server-side rate or bot controls. Listing a private path in robots.txt can reveal that the path exists while doing nothing to stop a non-compliant requester.
For article previews, decide what anonymous visitors and crawlers may actually receive. If the server sends the complete article in HTML and hides it only with front-end code, robots.txt cannot repair that access-control mistake.
A robots.txt file applies only to the same scheme, host, and port. If your publication uses separate hosts for the main site, archive, image CDN, or subscriber platform, each host needs its own policy. Use path-specific rules only when they reflect a stable content structure; broad root-level decisions are easier to audit than dozens of fragile exceptions.
/robots.txt on the exact host.For a ready starting point, use the allow-search, block-training template. The broader AI scraping limits guide explains when server-side enforcement is required.
Not automatically. Separate training, AI-search, user-request, and ordinary search agents, then block only the purposes that conflict with your publishing policy.
No. Paid or private content requires authentication and server-side authorization. Robots.txt is a public instruction for compliant crawlers, not an access-control system.
Crawler-specific training blocks should not affect Googlebot or Bingbot when those search agents remain allowed. A broad wildcard block can damage normal search crawling.
Yes when a provider publishes separate crawler controls for those purposes. Write independent groups for the documented tokens and verify each one separately.
Yes. Robots.txt rules apply only to the same scheme, host, and port. A separate archive, image host, or subscriber subdomain needs its own policy.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
robots.txt for AI Training: Build a Selective Opt-OutTo restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.
robots.txt for AI Search Visibility and CitationsTo support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.