How to Check Whether AI Crawlers Are Blocked

Fetch the live robots.txt file, select the crawler’s exact product token, and test the specific URL against the matching group and longest Allow or Disallow rule. Then check HTTP responses, caching, firewall rules, and logs, because a crawler may be allowed by robots.txt yet blocked by the server—or disallowed by robots.txt while an unidentified scraper still reaches the page.

Independent technical reference Updated Editorial approach Report a correction
Diagnostic flow checking a crawler token, URL rule match, server response, and access logs

Copy-paste robots.txt example

User-agent: GPTBot
Disallow: /research/
Allow: /research/public-summary.html

User-agent: OAI-SearchBot
Allow: /

Start with the live file, not a saved copy

Open https://your-domain.com/robots.txt for the exact host you want to test. Rules are scoped by protocol, host, and port, so a file on www.example.com does not automatically control shop.example.com. Confirm that the response is plain text and that a CDN, CMS, or security plugin is not serving a different file from the one you edited.

Use the exact crawler token

AI providers may publish separate product tokens for training, search discovery, and user-triggered requests. Test the exact token instead of assuming that one company name covers every agent. Examples include GPTBot and OAI-SearchBot, or ClaudeBot and Claude-SearchBot.

A crawler-specific group normally takes precedence over the wildcard group for that token. If the same exact token appears in more than one group, the protocol requires the matching groups to be combined rather than treating only the first group as active.

Test a real URL, not only the homepage

Permission is determined for each URL. The most specific matching path wins, and an equally specific Allow rule should win over Disallow. This means a site can block a directory while reopening one file:

User-agent: GPTBot
Disallow: /research/
Allow: /research/public-summary.html

Test both URLs. A result for / does not prove that /research/report.pdf has the same policy.

Use the checker systematically

  1. Enter the domain in the robots.txt checker.
  2. Confirm the fetched URL and HTTP status.
  3. Select or enter the exact crawler token.
  4. Test the exact page path, including relevant query parameters when needed.
  5. Review the matched group and the rule with the longest match.
  6. Repeat for training, AI-search, user-triggered, and regular search agents that matter to your policy.

Separate robots.txt from network blocking

A robots.txt result answers whether a compliant crawler should request a URL. It does not prove that the web server will accept the request. A firewall, bot-management service, rate limiter, authentication rule, CDN challenge, or application policy can return 403, 429, or another response even when robots.txt allows access.

The reverse is also possible: robots.txt may disallow the named bot, but a client that ignores the protocol can still receive a normal 200 response. Check access logs and security events when you need to understand actual traffic.

Account for caching and identity

Crawlers commonly cache robots.txt, so a new rule may not be reflected immediately. Do not repeatedly change the file while testing; publish one correct version and allow time for refresh. User-Agent strings can also be spoofed. If you are validating actual provider traffic, use the operator’s published verification method or IP information where available instead of trusting the header alone.

Final verification checklist

  • The correct host serves the intended robots.txt file.
  • The exact crawler token has been tested.
  • The target URL’s longest matching rule is understood.
  • No later firewall or application control contradicts the intended policy.
  • Logs are reviewed when actual traffic, not only declared policy, matters.

For pattern details, see robots.txt wildcards. For broader restrictions, use How to Block AI Crawlers.

FAQ

How do I test one AI crawler against one page?

Fetch the live robots.txt file, select the crawler’s exact product token, and evaluate the full target path against the applicable group and its longest matching rule.

Why is a bot blocked even though robots.txt says Allow?

A CDN, WAF, rate limiter, authentication rule, or application policy may deny the request after robots.txt. Check the actual HTTP response and security logs.

Why can a scraper access a URL that robots.txt disallows?

Robots.txt is voluntary. A client that ignores the protocol can still request a public URL unless the server enforces a restriction.

Do crawler-specific rules replace User-agent: * rules?

For a crawler with a matching specific group, that group is used instead of the wildcard group. Repeated groups matching the same product token are combined.

How quickly do new robots.txt rules take effect?

It depends on when each crawler refreshes its cached copy. Many crawlers cache the file, so publish the correct version and allow time before judging real traffic.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Robots.txt checker inspecting a website and reporting allowed and blocked crawler access robots.txt Checker

Enter the exact website address, choose a crawler, and test a path. The checker reads the live robots.txt file and explains the group and rule that determine access.

 Website selectively blocking AI training crawler routes while keeping search discovery routes open How to Block AI Crawlers with robots.txt

Use the exact published User-agent name for each crawler you intend to restrict, add Disallow: / to that group, preserve any search or user-requested access you still want, then publish and test the live file. Do not use User-agent: * unless you intend to restrict ordinary search crawlers too.

 Root-level crawler policy directing public and blocked website paths while private content uses separate server-side security robots.txt Definition, Syntax, and Examples

Robots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.

 Crawler identities matching specific robots.txt rule groups while unmatched agents follow a separate fallback group User-Agent in robots.txt: Definition and Matching

In robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.

 URL paths passing through wildcard, exact-ending, and longest-match robots.txt rule selection robots.txt Wildcards: * and $ Pattern Matching

In robots.txt path rules, * matches zero or more characters and $ anchors the match to the end. The most specific applicable rule decides the result, not the last line in the file. Build the complete policy, then test exact crawler and URL combinations before publishing.