Allow explicitly permits the matching crawler to request a path. It is most useful as a narrow exception inside a broader Disallow rule. The longest matching path wins; if equally specific Allow and Disallow rules conflict, Allow is preferred.
User-agent: * Disallow: /downloads/ Allow: /downloads/manuals/
Allow is a robots.txt rule that identifies URL paths a matching crawler may access. Most sites do not need Allow: / to make public content crawlable because access is unrestricted when no matching Disallow rule applies. Allow becomes valuable when a specific resource must remain open inside a directory or pattern that is otherwise blocked.
Robots.txt evaluates every matching Allow and Disallow path in the selected user-agent group. The most specific match wins. Specificity is based on the number of matched path bytes, not on which rule appears first in the file. A longer Allow path can therefore reopen a file or subdirectory inside a shorter Disallow path.
In the example below, /downloads/ is broadly excluded, but the longer /downloads/manuals/ path remains crawlable. Reversing the line order would not change the result because path specificity—not file order—determines the applicable rule.
If an Allow and a Disallow rule match the same path with equal specificity, the Robots Exclusion Protocol says Allow should be used. Avoid designing a policy around ties when a clearer, more specific path can express the same intent.
An empty Allow: line is ignored. It does not reopen the site and is not a shorthand for allowing everything. Either omit unnecessary rules or use a real path such as Allow: / when an explicit statement improves readability.
Before using an exception, ask whether the broad Disallow rule is itself too aggressive. A simpler set of narrow blocks is often easier to maintain than a large blocked area followed by many Allow repairs.
Allow paths are relative to the host root and are case-sensitive. The standard supports * as a multi-character wildcard and $ as an end marker. A trailing slash distinguishes a directory prefix from similarly named paths. Test representative URLs, including query strings and capitalization variants, before publishing a pattern.
The directive means “this compliant crawler may request this path under the robots policy.” It does not grant a user account permission, bypass a firewall, or make private content public. Server-side authentication and authorization always decide whether a request can actually access protected content.
No. Crawling is allowed by default when no matching Disallow rule applies. Allow: / can be used for clarity but is usually optional.
Yes, when the Allow path is the more specific match for the requested URL.
No. Robots.txt uses the most specific matching path, not the order of the lines in the file.
The Robots Exclusion Protocol says the Allow rule should be used when the matching rules are equivalent.
No. It only expresses crawler permission within robots.txt. Authentication and server-side authorization still control real access.
robots.txt Definition, Syntax, and ExamplesRobots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.
User-Agent in robots.txt: Definition and MatchingIn robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.
Disallow in robots.txt: Meaning and ExamplesDisallow tells the matching crawler not to request URL paths beginning with the specified pattern. Disallow: / blocks the entire host for that group, while an empty Disallow: rule is ignored. When rules conflict, the most specific matching path wins.
AI Robots.txt GeneratorChoose a policy mode, enter your website and sitemap, add any path or crawler overrides, then generate and download a robots.txt file. Publish it at the root of the correct host and verify the live rules with the checker before relying on them.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
