What Does Disallow Mean in robots.txt?

Disallow definition

Disallow is a robots.txt rule that asks the crawlers in the current User-agent group not to access matching URL paths. The value begins with a path relative to the root of the host. For example, Disallow: /private/ matches URLs whose path starts with /private/.

The rule affects crawling, not authorization. It does not hide the path, encrypt the content, or prevent a non-compliant client from requesting it.

Important Disallow values

Disallow: / is the broadest possible block because every path starts at the root. Use it only when the selected crawler should not crawl anything on that host. By contrast, an empty Disallow: line has no path and is ignored; it does not block the site.

Path matching starts at the beginning of the URL path and is case-sensitive. Therefore, /Private/ and /private/ can be different locations. A plain prefix may also match more than expected: Disallow: /fish can match /fish, /fish.html, and /fishheads. Add a trailing slash or an end marker when the intended boundary needs to be narrower.

Wildcards and exact endings

Compliant crawlers support * for zero or more characters and $ for the end of a match. For example, Disallow: /*.pdf$ targets paths ending in .pdf but not a URL where .pdf is followed by a query string. Test patterns against real URLs because one extra slash, capital letter, or query parameter can change the result.

Disallow and Allow precedence

When several rules match a URL, the most specific path wins—the one matching the greatest number of path bytes. This lets a narrow Allow exception reopen one public resource inside a broadly disallowed directory. If equally specific Allow and Disallow rules conflict, the Robots Exclusion Protocol prefers Allow.

The example below blocks a private directory but permits one explicitly public guide inside it. It is clearer and safer than adding multiple overlapping broad rules.

Crawling, indexing, and security

A disallowed URL may still be known to a search engine through links, sitemaps, or previous crawls. Because the crawler cannot fetch the page, it may also be unable to read a page-level noindex tag. If removal from search results is required, allow the search crawler to access the page long enough to process noindex or use the engine’s removal tools.

Never list genuinely secret locations as a substitute for access control. Robots.txt is public and can make hidden-looking paths easier to discover. Protect sensitive resources with authentication and server-side authorization.

Disallow checklist

• Target the smallest practical path instead of blocking the whole host.
• Check capitalization, trailing slashes, wildcards, and query strings.
• Test a permitted URL and a blocked URL after every change.
• Use Allow only for precise exceptions.
• Use the checker before publishing production rules.