robots.txt Wildcards: * and $ Rules, Examples, and Testing

In robots.txt path rules, * matches zero or more characters and $ anchors the match to the end. The most specific applicable rule decides the result, not the last line in the file. Build the complete policy, then test exact crawler and URL combinations before publishing.

Independent technical reference Updated August 1, 2026 Editorial approach Report a correction

Copy-paste robots.txt example

User-agent: *
Disallow: /*.pdf$
Disallow: /search?*sort=
Disallow: /private/*/draft/
Allow: /private/*/draft/public-summary.html$

What the wildcard characters actually do

Robots.txt supports two special characters inside Allow and Disallow path rules. The asterisk * matches zero or more characters, including slashes. The dollar sign $ marks the end of the match pattern. Together they can target file types, changing path segments, URL parameters, or one exact ending.

These operators are deliberately limited. Robots.txt does not provide a full regular-expression language, so constructs such as character classes, alternation, capture groups, lookarounds, and a single-character wildcard are not available.

Do not confuse a wildcard inside a path with User-agent: *. In a User-agent line, the asterisk identifies the fallback group for crawlers that do not have a more specific matching group. Inside an Allow or Disallow value, it matches characters in the requested URL.

Quick pattern reference

Rule path	What it matches	Important boundary
`/`	The root and every lower-level URL.	Use carefully: `Disallow: /` blocks the whole host for that crawler group.
`/*`	Equivalent to `/` for Google; the trailing wildcard adds nothing.	Do not use extra wildcards when a literal prefix already expresses the rule.
`/fish`	Any path beginning with `/fish`, including `/fish.html` and `/fishheads`.	Add a slash or `$` when that broader prefix is not intended.
`/fish$`	The exact path ending at `/fish`.	It does not match `/fish/`, `/fish.html`, or a parameterized variant.
`/*.pdf`	Any URL containing `.pdf` after a slash.	It can also match suffixes followed by parameters or more path text.
`/*.pdf$`	A URL whose match ends exactly with `.pdf`.	It does not match `/report.pdf?download=1`.
`/search?*sort=`	A search URL where characters occur between the literal query marker and `sort=`.	The question mark is literal URL syntax here; it is not a one-character wildcard.

Use * for variable text, not automatically for every prefix

The asterisk is useful when the changing part of a URL can occur between two stable parts of the pattern:

User-agent: *
Disallow: /private/*/draft/

This can match paths such as /private/team-a/draft/ and /private/2026/client/draft/. Because * can match zero characters as well, it can also affect a URL where the two fixed portions are adjacent.

A wildcard at the end of an ordinary prefix is often redundant. For Google, /fish* is equivalent to /fish. Both begin matching as soon as the URL starts with that text. Prefer the shorter literal rule unless the wildcard is needed between stable segments.

Use $ when the match must stop at an exact ending

Without the end marker, a path rule continues to match URLs that contain additional text after the shown pattern. Compare these two rules:

User-agent: *
Disallow: /*.pdf

User-agent: ExampleBot
Disallow: /*.pdf$

The first pattern can match /report.pdf, /report.pdf?download=1, and even a longer path containing .pdf. The second pattern is anchored and is intended for URLs ending exactly in .pdf.

This distinction matters when a site uses parameters for downloads, language selection, previews, sorting, or tracking. Decide whether the parameterized form should be blocked too, then test that exact URL instead of assuming the suffix rule covers it.

Wildcards can match URL parameters

Google's robots.txt examples include parameterized URLs in path matching. A rule can therefore target a stable query pattern:

User-agent: *
Disallow: /search?*sort=
Disallow: /*?sessionid=

The first rule targets search URLs that contain sort= after the shown prefix. The second can target URLs whose query begins with the displayed session parameter pattern.

Parameter order is a common source of mistakes. A URL such as /search?sort=price&page=2 is not textually identical to /search?page=2&sort=price. Test every parameter order your application actually produces. Where possible, fix duplicate or crawl-heavy parameter behavior at the application and canonicalization level instead of maintaining an increasingly complicated wildcard file.

The most specific matching rule decides the result

Applicable Allow and Disallow rules are not processed as a simple top-to-bottom firewall. The crawler selects the most specific matching path—the match with the greatest number of octets. When an equivalent Allow and Disallow rule conflict, Allow should be used.

User-agent: ExampleBot
Disallow: /media/
Allow: /media/public/
Disallow: /media/public/raw/*

For /media/public/logo.svg, the longer Allow rule is more specific than Disallow: /media/, so the URL is allowed. For /media/public/raw/source.svg, the longer raw-directory rule is the most specific match, so the URL is disallowed.

Moving the broad rule below the narrow rule does not reverse that result. Rule order is useful for human readability, but it is not a replacement for correct specificity.

Build narrow exceptions instead of broad repairs

Wildcards are often used to block a broad class of URLs and then reopen a smaller public set:

User-agent: *
Disallow: /*.pdf$
Allow: /public/*.pdf$

This requests that compliant crawlers avoid PDF URLs generally while allowing PDF files under /public/. Before publishing, test:

/private/report.pdf — expected blocked;
/public/report.pdf — expected allowed;
/public/report.pdf?download=1 — determine the intended result explicitly;
/public/report.PDF — path matching is case-sensitive;
/public/archive/report.pdf/preview — does not end at the PDF suffix.

If a broad block requires many Allow exceptions, the policy may be too difficult to maintain. Several literal Disallow rules can be safer than one large wildcard rule followed by a long exception list.

Path case and encoding can change the match

Allow and Disallow path values should be compared case-sensitively. A rule for /Private/ may not match /private/. This is especially important on sites where the application redirects letter-case variants or where legacy URLs use mixed case.

Percent-encoding also matters. RFC 9309 defines how non-ASCII, reserved, and percent-encoded characters are normalized for matching. If the actual URL contains a literal asterisk or dollar sign, encode the special character in the robots.txt pattern—%2A for * and %24 for $—so it is treated as URL data rather than an operator.

Always test the public URL produced by the browser or application. A CMS route name, database slug, or decoded development value may not be the exact string the crawler requests.

Patterns that are not regular expressions

The following regex-style ideas do not have their usual regular-expression meaning in robots.txt:

[0-9] does not define a numeric character class;
(draft|preview) does not create alternation;
.+ does not mean one or more characters;
? is not a generic one-character wildcard;
lookahead and lookbehind syntax is not supported.

Use separate literal rules when several distinct patterns must be controlled. A longer file with clear rules is generally easier to verify than a compact expression that the crawler does not interpret as intended.

Test a decision matrix before publishing

Do not validate only one URL that is expected to be blocked. Prepare both positive and negative cases for every important wildcard:

Test category	Example	Why it matters
Exact intended match	`/public/report.pdf`	Confirms the primary rule outcome.
Neighboring non-match	`/public/report.html`	Detects a pattern that is broader than intended.
Query variant	`/public/report.pdf?download=1`	Shows whether `$` and parameters change the result.
Case variant	`/Public/report.PDF`	Checks case-sensitive path behavior.
Nested path	`/public/archive/report.pdf`	Confirms whether the wildcard crosses directory levels.
Different crawler	The same URL for Googlebot and a named AI crawler	Verifies that the intended User-agent group is selected.

Use the robots.txt Checker to test each live crawler-and-URL combination. Inspect the selected User-agent group and the winning rule, not only the final Allowed or Blocked label.

Use Generator and Checker as one workflow

Define the outcome. List the URL forms that should be allowed and disallowed for each crawler.
Start with literal paths. Add * or $ only where the real URL structure requires it.
Review the complete file. A correct isolated wildcard can still conflict with another group or path rule.
Assemble the policy. Use the robots.txt Generator to build the crawler groups and path rules, then merge them with the site's existing robots.txt content.
Publish on the exact host. The file must be available at the root-level /robots.txt URL for the same scheme, host, and port.
Verify the live response. Open the public file, then test the full URL matrix in the Checker.

For the underlying rule behavior, review Disallow, Allow, and User-agent matching. The common robots.txt mistakes guide covers broader deployment and indexing errors.

Remember what robots.txt cannot do

A wildcard rule controls crawling only for clients that choose to follow robots.txt. It does not authenticate a crawler, stop a spoofed user-agent, secure private data, remove a URL from an index, or guarantee that an allowed URL will be crawled or ranked.

Use authentication and server-side authorization for private or paid content. Use appropriate indexing controls when the goal is to prevent search indexing. Treat wildcard rules as a precise crawl-management layer, not as a security boundary.

Protocol references

The definitions of *, $, case-sensitive path matching, equivalent Allow preference, and longest-match behavior are specified in RFC 9309. Google provides additional examples for prefixes, extensions, query strings, and end-anchored patterns in its robots.txt specification documentation.

Next step: assemble the complete policy in the Generator, publish it, and test every important wildcard boundary in the live Checker.

FAQ

What does * mean in robots.txt?

Inside an Allow or Disallow path, * matches zero or more characters, including slashes. It is different from User-agent: *, which identifies the fallback crawler group.

What does $ mean in robots.txt?

The dollar sign marks the end of the match pattern. For example, /*.pdf$ targets a URL ending exactly in .pdf and does not match a longer query-string variant such as /report.pdf?download=1.

Is /* different from / in robots.txt?

For Google, /* is equivalent to /. A trailing wildcard after a prefix usually adds nothing, so the shorter literal rule is easier to read and maintain.

Does the last matching robots.txt rule win?

No. The most specific matching path is selected. If equivalent Allow and Disallow rules conflict, Allow should be used. Visual order does not override specificity.

Are robots.txt wildcards regular expressions?

No. Robots.txt defines only limited special characters such as * and $. Character classes, alternation, capture groups, lookarounds, and other regex syntax are not supported as regular expressions.

Are robots.txt paths case-sensitive?

They should be matched case-sensitively. Test uppercase and lowercase variants that your site actually serves, along with encoded and parameterized URLs.

How do I match a literal * or $ in a URL?

Percent-encode the character in the robots.txt pattern so it is treated as URL data: use %2A for a literal asterisk and %24 for a literal dollar sign.

How should I test a wildcard rule?

Test the exact intended match, a neighboring non-match, a query-string variant, a case variant, a nested path, and the same URL for every relevant crawler. Use the live Checker and inspect the selected group and winning rule.