Robots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.
User-agent: * Allow: / Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Robots.txt is the standard file site owners use to communicate crawl-access preferences to automated clients such as search engine crawlers. The formal name is the Robots Exclusion Protocol. A compliant crawler fetches the file before requesting other URLs on the same host, selects the group that matches its product token, and applies the group’s path rules.
The file is a request to crawlers, not an authentication system. A bot can ignore it, and every path listed in the file is publicly visible. Confidential, paid, or account-only content still needs real server-side access control.
The filename and location are fixed: lowercase /robots.txt at the top level of the service, such as https://example.com/robots.txt. It must be served as plain text and encoded as UTF-8. Rules are scoped to the same scheme, host, and port. Therefore, https://example.com/robots.txt does not automatically control https://shop.example.com/.
A normal successful response lets the crawler parse and follow the valid rules. Redirects, missing files, and server failures can be handled differently by crawlers, so the file should return a stable response and should not be generated by an unreliable application route.
A group starts with one or more User-agent lines followed by rules. Disallow identifies path prefixes the matching crawler should not request. Allow identifies paths it may request, most often as a specific exception inside a broader block. If more than one path rule matches, the most specific match—the rule matching the greatest number of path bytes—wins. When equally specific Allow and Disallow rules conflict, the standard prefers Allow.
The wildcard product token * is the fallback group for crawlers without a more specific match. Repeated groups for the same product token are combined by compliant implementations. Path values are case-sensitive even though field names and user-agent matching are case-insensitive.
Robots.txt controls crawling. It does not guarantee that a known URL disappears from a search index, and it does not prevent a person or non-compliant scraper from opening the URL. For search removal, use an accessible noindex directive or the relevant search-engine removal tool. For privacy or authorization, require authentication and enforce permissions on the server.
The file may also contain Sitemap records. Major search engines use these absolute URLs to discover sitemap files, but Sitemap is not an Allow or Disallow rule and is not tied to one crawler group.
The example keeps public content crawlable, excludes a private directory, and advertises a sitemap. Replace the sample paths and hostname before publishing it. Then test both allowed and blocked URLs with the robots.txt checker.
Disallow: / unintentionally and blocking the whole site.It must be available at the lowercase top-level path /robots.txt for the exact host, such as https://example.com/robots.txt.
No. It is a public instruction file for compliant crawlers. Protect private content with authentication, authorization, and server-side access checks.
Not reliably. It can stop crawling, but a search engine may still know the URL. Use noindex on an accessible page or a search-engine removal tool for index exclusion.
The most specific matching path rule wins. If equally specific Allow and Disallow rules conflict, the Robots Exclusion Protocol prefers Allow.
No. The policy is scoped to the exact scheme, host, and port, so each subdomain needs its own robots.txt file.
User-Agent in robots.txt: Definition and MatchingIn robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.
Disallow in robots.txt: Meaning and ExamplesDisallow tells the matching crawler not to request URL paths beginning with the specified pattern. Disallow: / blocks the entire host for that group, while an empty Disallow: rule is ignored. When rules conflict, the most specific matching path wins.
Allow in robots.txt: Meaning and PriorityAllow explicitly permits the matching crawler to request a path. It is most useful as a narrow exception inside a broader Disallow rule. The longest matching path wins; if equally specific Allow and Disallow rules conflict, Allow is preferred.
AI Robots.txt GeneratorChoose a policy mode, enter your website and sitemap, add any path or crawler overrides, then generate and download a robots.txt file. Publish it at the root of the correct host and verify the live rules with the checker before relying on them.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.