robots.txt Checker: Test Live and Draft Rules

Choose a live test or a draft simulation

Use the live check to inspect a site’s published robots.txt response and its exact rule for one crawler and path. Use the draft test to paste an unpublished policy and compare up to eight paths before you change the server. The useful question in both modes is which group and rule actually win for the crawler and path you selected.

• Test the homepage and one restricted path. A rule can behave correctly at / and still fail on a deeper URL.
• Test the crawler that matters. A specific User-agent group can produce a different result from User-agent: *.
• Read the HTTP result first. A 404, rate limit, server error, redirect problem, or HTML response is not the same as a valid robots.txt policy.
• Repeat the test after deployment. This checker reads the live response, so it can confirm what the public server actually returns.

For a new policy, create the draft in the Generator, simulate its paths before publication, merge it with existing live rules, then verify the published file with the live checker.

What the live and draft checks test

The checker requests the public /robots.txt file for the website address you enter, then evaluates the exact path against a selected crawler name. The result shows the HTTP response, applicable User-agent group, most specific matching rule, and a plain-language access verdict.

The live mode requests the site’s public robots.txt file; draft mode parses the rules you enter without fetching any website. Neither publishes a replacement file, checks private routes or authenticates bots. The draft is not stored by this tool. Run the live check again after deployment to confirm the server’s actual response.

Test the exact scheme and host

Robots.txt applies to the origin that serves it. Rules at https://example.com/robots.txt do not automatically describe http://example.com, https://www.example.com, or https://shop.example.com. Enter the same website address that users and crawlers reach.

When you include http:// or https://, the checker tests that scheme. When you enter only a domain, it tries HTTPS first and uses HTTP only if HTTPS cannot be reached. The result always shows the final robots.txt URL and any redirects followed.

Check the HTTP response before interpreting rules

A robots.txt policy can be evaluated only when the server returns a usable text response. The checker does not turn an error page, rate-limit response, or server failure into a misleading allow or block verdict.

• Successful 2xx response: the returned text is parsed and tested.
• 4xx response: no usable file was returned. Many compliant crawlers treat most 4xx responses as no robots restrictions, but provider-specific behavior can differ.
• HTTP 429: the request was rate limited, so the response does not establish the live policy.
• 5xx or network failure: the file was temporarily unreachable. Crawlers may react conservatively or use a cached copy.
• HTML or oversized response: the response is not treated as a valid robots.txt text file.

If the live status is unexpected, fix availability, redirects, caching, or response headers before editing crawler rules. A correct draft cannot help when the public URL serves the wrong response.

How the matching User-agent group is selected

The selected crawler name is compared with the User-agent groups in the file. When an exact crawler-specific group exists, that group is used instead of User-agent: *. The exact group and wildcard group are not automatically combined.

User-agent: ExampleBot
Allow: /

User-agent: *
Disallow: /members/

For ExampleBot, the exact group applies. The wildcard /members/ rule is not inherited. This is why a crawler may appear allowed even when the wildcard group blocks the same path for other crawlers.

How the winning path rule is chosen

After selecting the group, the checker compares the tested path with its Allow and Disallow rules. The longest matching path is the most specific and normally wins. When equally specific Allow and Disallow rules both match, Allow takes precedence.

Test the real request path, including a query string when it matters. Fragments beginning with # are not sent to servers and do not belong in a robots.txt path test. Wildcards such as * and end anchors such as $ can change which rule wins, so test representative URLs rather than only the homepage.

What Allowed and Blocked actually mean

Allowed means that the fetched robots.txt file does not ask the selected crawler to avoid the tested path. It does not guarantee that the crawler will visit, index, rank, cite, or use the page.

Blocked means that the most specific applicable Disallow rule covers the tested path. It remains a voluntary crawling instruction. A non-compliant client can ignore it, and a claimed user-agent name does not authenticate the requester.

Robots.txt is not a privacy or security boundary. Protect customer information, paid content, private files, administration areas, and confidential APIs with authentication, authorization, server rules, and application-level controls.

Use the warnings as a review list

The result highlights malformed or unsupported lines and warns when a regular search crawler is blocked for the tested path. A warning is not automatically a defect: the correct decision depends on the site and the intended policy.

A missing Sitemap line is shown as information rather than an error because the directive is optional. A sitemap can also be submitted directly through search-engine tools or discovered by other supported methods.

Worked draft test: a protected directory with a public exception

Paste the following sample into Test an unpublished robots.txt draft, select GPTBot and enter /, /private/account, and /private/public/guide as three separate paths.

User-agent: *
Disallow: /private/
Allow: /private/public/

This result is a policy simulation only. A server firewall can still deny an allowed path, and a client ignoring robots.txt can still request a disallowed one. The draft evaluator does not access or publish any of these URLs.

A safe workflow after the check

1. Confirm the exact robots.txt URL, HTTP status, and response type.
2. Choose the crawler that matters to your policy instead of assuming one rule covers every provider use.
3. Test important public, private, search, product, and download paths separately.
4. Review the group used and the winning rule, not only the Allowed or Blocked badge.
5. Save the current live file before editing it.
6. Merge changes carefully, publish them at the correct host root, and repeat the live check.
7. Use server logs and provider verification methods when you need evidence of real crawler requests.

When the current file does not match your intended policy, use the generator for a reviewable draft, then merge it with any existing platform, search, and sitemap rules instead of replacing the live file blindly.