In robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.
User-agent: Bingbot Disallow: /internal-search/ User-agent: * Allow: /
The User-agent field starts a robots.txt rule group and identifies which crawler should follow that group. The value is usually a short product token such as Bingbot, GPTBot, or Googlebot. It is not necessarily the crawler’s complete HTTP User-Agent header. The product token should appear within the longer identification string sent with the request.
Field names and product-token matching are case-insensitive. Path values in the Allow and Disallow rules beneath the group remain case-sensitive.
A crawler first looks for a group matching its own product token. If a specific match exists, that group supplies the applicable rules. If no specific group matches, the crawler uses User-agent: * when that fallback group is present. When neither a specific group nor a wildcard group exists, no robots.txt restrictions apply to that crawler.
Do not assume that rules from a specific crawler group are automatically added to the wildcard group. If you create a dedicated Bingbot or GPTBot section, include all rules that agent should follow. This prevents an important global restriction from disappearing when the more specific group is selected.
One group can apply to several crawlers by placing multiple consecutive User-agent lines before the rules. This is useful when different agents should follow exactly the same policy. Use one line per product token rather than inventing a comma-separated list unless you have verified that every target crawler supports that extension.
If the same product token appears in more than one group, the Robots Exclusion Protocol requires matching groups to be combined. Keeping one clear group per token is still easier to review and less likely to create accidental conflicts.
User-agent: * means all crawlers that do not have a more specific matching group. It is useful for a site-wide default policy. It does not mean “merge these rules into every named group.” A common pattern is a permissive wildcard group for ordinary crawling plus separate groups for agents that need different access.
Robots.txt matching relies on the declared product token, but a malicious client can copy a well-known HTTP User-Agent string. Do not treat the name alone as proof that traffic comes from Microsoft, Google, or another operator. When identity matters, use the verification method published by that crawler operator, such as reverse-and-forward DNS or an official IP-range list.
The example gives Bingbot a specific restriction while allowing unmatched crawlers. Before publishing, verify the exact token in the crawler operator’s documentation and test representative URLs with the checker.
* only as a fallback policy.No. Robots.txt field names and crawler product-token matching are case-insensitive, although Allow and Disallow path values are case-sensitive.
It is the fallback group for crawlers that do not have a more specific matching product-token group.
Do not rely on that. A specifically matching group takes precedence over the wildcard fallback, so the specific group should contain every rule that crawler needs.
Yes. Place multiple consecutive User-agent lines before the shared Allow and Disallow rules, using one documented product token per line.
Under the Robots Exclusion Protocol, the rules from matching groups for the same product token are combined before evaluation.
robots.txt Definition, Syntax, and ExamplesRobots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.
Disallow in robots.txt: Meaning and ExamplesDisallow tells the matching crawler not to request URL paths beginning with the specified pattern. Disallow: / blocks the entire host for that group, while an empty Disallow: rule is ignored. When rules conflict, the most specific matching path wins.
Allow in robots.txt: Meaning and PriorityAllow explicitly permits the matching crawler to request a path. It is most useful as a narrow exception inside a broader Disallow rule. The longest matching path wins; if equally specific Allow and Disallow rules conflict, Allow is preferred.
Bingbot robots.txt Rules for Bing SearchBingbot is Microsoft Bing’s primary search crawler. It follows Bingbot-specific robots.txt groups and supports Crawl-delay. Blocking a path limits crawling but does not reliably remove an already known URL from Bing, so use page-level noindex or Bing removal tools when exclusion from search results is the real goal.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.