Does robots.txt Stop AI Scraping? What It Can and Cannot Do

Robots.txt can stop compliant AI crawlers from requesting selected URLs, but it cannot enforce access against unidentified scrapers, spoofed user agents, browsers, or tools that ignore the protocol. Use precise crawler rules for policy, then add authentication, authorization, rate limits, or firewall controls where access must actually be prevented.

Independent technical reference Updated Editorial approach Report a correction
Voluntary robots.txt crawler rules shown separately from enforceable server-side access controls

Copy-paste robots.txt example

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Googlebot
Allow: /

Robots.txt is a request, not an enforcement wall

A robots.txt file tells compliant automated clients which URLs they should not request. Major operators publish crawler tokens and state that their automated bots respect site rules, so a precise Disallow group can meaningfully reduce legitimate AI crawling. The protocol itself, however, does not authenticate the requester, hide the listed URLs, or stop a client that chooses to ignore the instruction.

This distinction matters because “AI scraping” can describe several different activities: scheduled training-data collection, AI-search discovery, a page fetch triggered by a user, an unidentified scraper, or a browser automation tool. One robots.txt rule does not control all of them in the same way.

What robots.txt can control

When a provider documents a dedicated token, you can publish a crawler-specific group without blocking unrelated traffic. For example, a site may block training-oriented crawlers while keeping conventional search and selected AI-search crawlers open:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Googlebot
Allow: /

This expresses a selective publishing policy. It can reduce access by compliant named bots and avoid the SEO damage caused by a blanket User-agent: * block.

What robots.txt cannot stop

  • Non-compliant scrapers: a client can ignore the file entirely.
  • Spoofed identities: any requester can send a familiar User-Agent string.
  • Human or browser access: a public URL remains public unless the server restricts it.
  • Previously obtained copies: a new rule does not recall content already collected.
  • Every user-triggered fetch: some providers distinguish automated crawlers from requests initiated by a person.

Robots.txt is also public. Listing a sensitive path there can reveal that the path exists, which is another reason not to treat the file as security.

Use enforcement controls for real restrictions

If a resource must not be fetched, place the restriction at the application or server layer. Depending on the case, that may mean login-based authentication, role-based authorization, signed URLs, a paywall, rate limiting, bot management, a web application firewall, or removing the resource from the public web. Avoid broad IP blocking unless the provider publishes verifiable ranges and you understand the risk of false positives.

For expensive public endpoints, combine a clear robots.txt policy with request limits and monitoring. For private content, do not publish it anonymously and hope crawlers comply.

Verify both policy and enforcement

  1. Open the live file at https://your-domain.com/robots.txt.
  2. Test the exact crawler token and target URL with the robots.txt checker.
  3. Check for a more specific Allow rule or a crawler-specific group that changes the result.
  4. Review server logs to distinguish a robots policy decision from a firewall or application denial.
  5. Use the AI crawler blocking guide when you need a selective configuration.

The practical rule is simple: use robots.txt to communicate crawler policy; use server-side controls to enforce access.

FAQ

Can robots.txt completely prevent AI companies from using my content?

No. It can restrict compliant crawlers that honor the published token, but it cannot enforce access against unidentified clients, copied data, browser automation, or scrapers that ignore the protocol.

Is blocking User-agent: * the safest option?

Usually not. A wildcard block can also stop Googlebot, Bingbot, AI-search crawlers, monitoring tools, and other legitimate clients. Target the exact crawler tokens that match your policy.

Can a scraper pretend to be Googlebot or an AI crawler?

Yes. User-Agent headers are self-declared and can be spoofed. For network enforcement, use provider verification methods where available and do not trust the header alone.

What should protect private or paid content?

Use authentication, authorization, signed access, paywall logic, server rules, or another enforceable control. Robots.txt should never be the only protection for sensitive resources.

Will a new Disallow rule remove content already collected?

No. It affects future requests by compliant crawlers after they refresh the file. It does not delete earlier copies or guarantee removal from a provider’s existing systems.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Website selectively blocking AI training crawler routes while keeping search discovery routes open How to Block AI Crawlers with robots.txt

Use the exact published User-agent name for each crawler you intend to restrict, add Disallow: / to that group, preserve any search or user-requested access you still want, then publish and test the live file. Do not use User-agent: * unless you intend to restrict ordinary search crawlers too.

 Website allowing an AI search discovery route while blocking a separate model-training route How to Allow AI Search but Block AI Training

Block only the documented training or product-use controls, such as GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended. Leave AI search crawlers unblocked unless an existing rule blocks them; add explicit search groups only when necessary, and repeat any private-path restrictions inside those groups. Then test the live file by crawler and path.

 Website blocking selected AI training and dataset routes while keeping search discovery open robots.txt for AI Training: Build a Selective Opt-Out

To restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.

 Public pages moving through AI search discovery and technical access toward possible source citation robots.txt for AI Search Visibility and Citations

To support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.

 Robots.txt checker inspecting a website and reporting allowed and blocked crawler access robots.txt Checker

Enter the exact website address, choose a crawler, and test a path. The checker reads the live robots.txt file and explains the group and rule that determine access.