A successful 2xx response lets crawlers parse robots.txt. Redirects may be followed, ordinary 4xx responses often mean no usable robots.txt restrictions, while 5xx and network failures are treated as temporary retrieval problems. Exact behavior and cache timing vary by crawler, so return a stable 200 response for a valid file and never use 401 or 403 as a substitute for crawl rules.
# Serve this content from /robots.txt with HTTP 200 and text/plain User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Crawlers must fetch /robots.txt before they can apply its rules. The HTTP result therefore matters as much as the file contents. A valid policy should normally be served as UTF-8 plain text with a stable 200 OK response.
A successful response tells the crawler that robots.txt was retrieved. It parses the valid groups and rules it understands. Returning HTML with a 200 status is still a mistake: some crawlers may extract parseable lines, but the result is unpredictable and difficult to maintain.
# Recommended response for /robots.txt
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
User-agent: *
Allow: /The Robots Exclusion Protocol recommends following at least five consecutive redirects. Google documents similar behavior and then treats an unresolved redirect chain like a missing file. Keep the final robots.txt URL simple, avoid loops, and remember that rules are applied to the original host for which the file was requested.
Under the standard, a 400–499 response means the robots.txt resource is unavailable and the crawler may access other resources. Google treats ordinary 4xx responses—except 429—as though no valid robots.txt file exists, so there are no crawl restrictions from that file.
If you want to block a crawler, return 200 with a valid Disallow rule. Do not protect robots.txt itself behind authentication.
The standard treats an unreachable robots.txt file, including server and network failures, conservatively and permits crawlers to use cached rules or temporarily assume broad restriction. Provider implementations add their own retry and cache policies.
Google documents a staged process: it initially pauses crawling while retrying, may use the last known good file for an extended period, and later changes behavior depending on whether the site and a cached copy remain available. A short outage can therefore preserve older rules rather than applying the file you intended to publish.
A hosting migration, CDN rule, maintenance page, redirect plugin, or security layer can silently change the robots.txt response. A 403 that looks restrictive to a human may actually remove crawl restrictions for some bots. A prolonged 5xx can pause crawling or keep stale rules in effect.
curl -I, or the robots.txt checker.For syntax after a successful fetch, review the robots.txt definition and wildcard matching.
Normally it should return 200 OK with a valid UTF-8 plain-text body. That gives crawlers an unambiguous policy to parse.
Not reliably. Google treats 403 like an unavailable robots.txt file and assumes no crawl restrictions. Return 200 with explicit Disallow rules instead.
It generally means no robots.txt file is available, so compliant crawlers may crawl without restrictions from that file.
They are temporary retrieval failures. Crawlers may pause, retry, or use a cached copy. Exact timing differs by provider, so restore a stable 200 response quickly.
Yes, crawlers may follow redirects, but only a limited chain is required. Avoid loops and excessive hops, and verify the final response and file contents.
robots.txt Definition, Syntax, and ExamplesRobots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
Common robots.txt Mistakes and How to Fix ThemThe most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.
How to Add a Sitemap to robots.txt CorrectlyAdd a standalone Sitemap line containing the full absolute URL of your XML sitemap or sitemap index, such as Sitemap: https://example.com/sitemap.xml. The line may appear anywhere in robots.txt and may be repeated, but it only helps discovery: it does not override Disallow rules or guarantee crawling or indexing.
robots.txt Wildcards: * and $ Pattern MatchingIn robots.txt, * matches zero or more characters and $ anchors the match to the end of the URL pattern. These are limited pattern operators, not full regular expressions. Crawlers compare applicable Allow and Disallow paths and use the most specific match; if equally specific rules conflict, Allow should win.