Fetch the live robots.txt file, select the crawler’s exact product token, and test the specific URL against the matching group and longest Allow or Disallow rule. Then check HTTP responses, caching, firewall rules, and logs, because a crawler may be allowed by robots.txt yet blocked by the server—or disallowed by robots.txt while an unidentified scraper still reaches the page.
User-agent: GPTBot Disallow: /research/ Allow: /research/public-summary.html User-agent: OAI-SearchBot Allow: /
Open https://your-domain.com/robots.txt for the exact host you want to test. Rules are scoped by protocol, host, and port, so a file on www.example.com does not automatically control shop.example.com. Confirm that the response is plain text and that a CDN, CMS, or security plugin is not serving a different file from the one you edited.
AI providers may publish separate product tokens for training, search discovery, and user-triggered requests. Test the exact token instead of assuming that one company name covers every agent. Examples include GPTBot and OAI-SearchBot, or ClaudeBot and Claude-SearchBot.
A crawler-specific group normally takes precedence over the wildcard group for that token. If the same exact token appears in more than one group, the protocol requires the matching groups to be combined rather than treating only the first group as active.
Permission is determined for each URL. The most specific matching path wins, and an equally specific Allow rule should win over Disallow. This means a site can block a directory while reopening one file:
User-agent: GPTBot
Disallow: /research/
Allow: /research/public-summary.htmlTest both URLs. A result for / does not prove that /research/report.pdf has the same policy.
A robots.txt result answers whether a compliant crawler should request a URL. It does not prove that the web server will accept the request. A firewall, bot-management service, rate limiter, authentication rule, CDN challenge, or application policy can return 403, 429, or another response even when robots.txt allows access.
The reverse is also possible: robots.txt may disallow the named bot, but a client that ignores the protocol can still receive a normal 200 response. Check access logs and security events when you need to understand actual traffic.
Crawlers commonly cache robots.txt, so a new rule may not be reflected immediately. Do not repeatedly change the file while testing; publish one correct version and allow time for refresh. User-Agent strings can also be spoofed. If you are validating actual provider traffic, use the operator’s published verification method or IP information where available instead of trusting the header alone.
For pattern details, see robots.txt wildcards. For broader restrictions, use How to Block AI Crawlers.
Fetch the live robots.txt file, select the crawler’s exact product token, and evaluate the full target path against the applicable group and its longest matching rule.
A CDN, WAF, rate limiter, authentication rule, or application policy may deny the request after robots.txt. Check the actual HTTP response and security logs.
Robots.txt is voluntary. A client that ignores the protocol can still request a public URL unless the server enforces a restriction.
For a crawler with a matching specific group, that group is used instead of the wildcard group. Repeated groups matching the same product token are combined.
It depends on when each crawler refreshes its cached copy. Many crawlers cache the file, so publish the correct version and allow time before judging real traffic.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
robots.txt Definition, Syntax, and ExamplesRobots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.
User-Agent in robots.txt: Definition and MatchingIn robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.
robots.txt Wildcards: * and $ Pattern MatchingIn robots.txt, * matches zero or more characters and $ anchors the match to the end of the URL pattern. These are limited pattern operators, not full regular expressions. Crawlers compare applicable Allow and Disallow paths and use the most specific match; if equally specific rules conflict, Allow should win.