Robots.txt can stop compliant AI crawlers from requesting selected URLs, but it cannot enforce access against unidentified scrapers, spoofed user agents, browsers, or tools that ignore the protocol. Use precise crawler rules for policy, then add authentication, authorization, rate limits, or firewall controls where access must actually be prevented.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Googlebot Allow: /
A robots.txt file tells compliant automated clients which URLs they should not request. Major operators publish crawler tokens and state that their automated bots respect site rules, so a precise Disallow group can meaningfully reduce legitimate AI crawling. The protocol itself, however, does not authenticate the requester, hide the listed URLs, or stop a client that chooses to ignore the instruction.
This distinction matters because “AI scraping” can describe several different activities: scheduled training-data collection, AI-search discovery, a page fetch triggered by a user, an unidentified scraper, or a browser automation tool. One robots.txt rule does not control all of them in the same way.
When a provider documents a dedicated token, you can publish a crawler-specific group without blocking unrelated traffic. For example, a site may block training-oriented crawlers while keeping conventional search and selected AI-search crawlers open:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /This expresses a selective publishing policy. It can reduce access by compliant named bots and avoid the SEO damage caused by a blanket User-agent: * block.
Robots.txt is also public. Listing a sensitive path there can reveal that the path exists, which is another reason not to treat the file as security.
If a resource must not be fetched, place the restriction at the application or server layer. Depending on the case, that may mean login-based authentication, role-based authorization, signed URLs, a paywall, rate limiting, bot management, a web application firewall, or removing the resource from the public web. Avoid broad IP blocking unless the provider publishes verifiable ranges and you understand the risk of false positives.
For expensive public endpoints, combine a clear robots.txt policy with request limits and monitoring. For private content, do not publish it anonymously and hope crawlers comply.
https://your-domain.com/robots.txt.Allow rule or a crawler-specific group that changes the result.The practical rule is simple: use robots.txt to communicate crawler policy; use server-side controls to enforce access.
No. It can restrict compliant crawlers that honor the published token, but it cannot enforce access against unidentified clients, copied data, browser automation, or scrapers that ignore the protocol.
Usually not. A wildcard block can also stop Googlebot, Bingbot, AI-search crawlers, monitoring tools, and other legitimate clients. Target the exact crawler tokens that match your policy.
Yes. User-Agent headers are self-declared and can be spoofed. For network enforcement, use provider verification methods where available and do not trust the header alone.
Use authentication, authorization, signed access, paywall logic, server rules, or another enforceable control. Robots.txt should never be the only protection for sensitive resources.
No. It affects future requests by compliant crawlers after they refresh the file. It does not delete earlier copies or guarantee removal from a provider’s existing systems.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
robots.txt for AI Training: Build a Selective Opt-OutTo restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.
robots.txt for AI Search Visibility and CitationsTo support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.