In robots.txt, * matches zero or more characters and $ anchors the match to the end of the URL pattern. These are limited pattern operators, not full regular expressions. Crawlers compare applicable Allow and Disallow paths and use the most specific match; if equally specific rules conflict, Allow should win.
User-agent: * Disallow: /*.pdf$ Disallow: /search?*sort= Disallow: /private/*/draft/ Allow: /private/*/draft/public-summary.html$
The Robots Exclusion Protocol defines two special pattern characters for Allow and Disallow paths. The asterisk * represents zero or more characters, and the dollar sign $ marks the end of the match. They are useful, but they do not turn robots.txt into a complete regular-expression language.
Do not confuse the path wildcard with User-agent: *. In the User-agent field, the asterisk selects crawlers that do not have a more specific matching group. In a path rule, it matches characters inside a URL pattern.
To block URLs that end in .pdf, anchor the suffix:
User-agent: *
Disallow: /*.pdf$The ending anchor prevents the rule from also matching a URL such as /file.pdf?download=1. Whether that query variant should also be blocked depends on your goal; test the real URL forms your site generates.
An asterisk can cover changing folders, filenames, or parameters:
User-agent: *
Disallow: /search?*sort=
Disallow: /private/*/draft/
Allow: /private/*/draft/public-summary.html$The first rule targets search URLs containing a sort parameter after the shown prefix. The second targets draft folders at varying path levels. The longer Allow rule reopens one exact public summary.
Crawlers compare all applicable Allow and Disallow paths and select the most specific match, measured by the length of the matched pattern. If two equivalent rules conflict, the protocol says Allow should be used.
User-agent: ExampleBot
Disallow: /media/
Allow: /media/public/
Disallow: /media/public/raw/Here, /media/public/logo.svg is allowed, while /media/public/raw/source.svg is disallowed by the longer rule. Rule order is not a reliable substitute for specificity.
Path comparison should be case-sensitive. A rule for /Private/ may not match /private/. Percent-encoded characters also require care, because the protocol defines how URI characters are encoded and compared. Test the final URLs rather than reasoning only from a CMS route name.
* as one character instead of zero or more.$ when only an exact suffix should match.Before publishing, prepare a small matrix of URLs that should be allowed and disallowed. Run each one through the robots.txt checker with the exact crawler token. Include boundary cases: the folder itself, nested pages, similar filenames, uppercase variants, parameters, and the exact suffix.
For the underlying fields, review Disallow, Allow, and User-Agent matching.
In an Allow or Disallow path, * represents zero or more characters. It is a wildcard, not a full regular-expression operator.
The dollar sign anchors the pattern to the end of the URL match. It is useful for exact filenames or suffixes such as /*.pdf$.
No. The most specific matching path is used. If equally specific Allow and Disallow rules conflict, Allow should win.
They should be compared case-sensitively. Test uppercase, lowercase, encoded characters, and query variants that your site actually serves.
No. The standard defines * and $ for path matching, but common regex constructs such as character classes, alternation, and lookarounds are not part of robots.txt syntax.
Disallow in robots.txt: Meaning and ExamplesDisallow tells the matching crawler not to request URL paths beginning with the specified pattern. Disallow: / blocks the entire host for that group, while an empty Disallow: rule is ignored. When rules conflict, the most specific matching path wins.
Allow in robots.txt: Meaning and PriorityAllow explicitly permits the matching crawler to request a path. It is most useful as a narrow exception inside a broader Disallow rule. The longest matching path wins; if equally specific Allow and Disallow rules conflict, Allow is preferred.
User-Agent in robots.txt: Definition and MatchingIn robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
Common robots.txt Mistakes and How to Fix ThemThe most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.