Check robots.txt Access for Any Crawler and Path

What this robots.txt checker tests

The checker requests the public /robots.txt file for the website address you enter, then evaluates the exact path against a selected crawler name. The result shows the HTTP response, applicable User-agent group, most specific matching rule, and a plain-language access verdict.

This is a live check, not a test of an unpublished draft. It does not change your website, inspect plugin settings, crawl other pages, or upload a replacement file. Run it again after deployment to confirm what the server is actually returning.

Test the exact scheme and host

Robots.txt applies to the origin that serves it. Rules at https://example.com/robots.txt do not automatically describe http://example.com, https://www.example.com, or https://shop.example.com. Enter the same website address that users and crawlers reach.

When you include http:// or https://, the checker tests that scheme. When you enter only a domain, it tries HTTPS first and uses HTTP only if HTTPS cannot be reached. The result always shows the final robots.txt URL and any redirects followed.

Check the HTTP response before interpreting rules

A robots.txt policy can be evaluated only when the server returns a usable text response. The checker does not turn an error page, rate-limit response, or server failure into a misleading allow or block verdict.

• Successful 2xx response: the returned text is parsed and tested.
• 4xx response: no usable file was returned. Many compliant crawlers treat most 4xx responses as no robots restrictions, but provider-specific behavior can differ.
• HTTP 429: the request was rate limited, so the response does not establish the live policy.
• 5xx or network failure: the file was temporarily unreachable. Crawlers may react conservatively or use a cached copy.
• HTML or oversized response: the response is not treated as a valid robots.txt text file.

If the live status is unexpected, fix availability, redirects, caching, or response headers before editing crawler rules. A correct draft cannot help when the public URL serves the wrong response.

How the matching User-agent group is selected

The selected crawler name is compared with the User-agent groups in the file. When an exact crawler-specific group exists, that group is used instead of User-agent: *. The exact group and wildcard group are not automatically combined.

User-agent: ExampleBot
Allow: /

User-agent: *
Disallow: /members/

For ExampleBot, the exact group applies. The wildcard /members/ rule is not inherited. This is why a crawler may appear allowed even when the wildcard group blocks the same path for other crawlers.

How the winning path rule is chosen

After selecting the group, the checker compares the tested path with its Allow and Disallow rules. The longest matching path is the most specific and normally wins. When equally specific Allow and Disallow rules both match, Allow takes precedence.

Test the real request path, including a query string when it matters. Fragments beginning with # are not sent to servers and do not belong in a robots.txt path test. Wildcards such as * and end anchors such as $ can change which rule wins, so test representative URLs rather than only the homepage.

What Allowed and Blocked actually mean

Allowed means that the fetched robots.txt file does not ask the selected crawler to avoid the tested path. It does not guarantee that the crawler will visit, index, rank, cite, or use the page.

Blocked means that the most specific applicable Disallow rule covers the tested path. It remains a voluntary crawling instruction. A non-compliant client can ignore it, and a claimed user-agent name does not authenticate the requester.

Robots.txt is not a privacy or security boundary. Protect customer information, paid content, private files, administration areas, and confidential APIs with authentication, authorization, server rules, and application-level controls.

Use the warnings as a review list

The result highlights malformed or unsupported lines and warns when a regular search crawler is blocked for the tested path. A warning is not automatically a defect: the correct decision depends on the site and the intended policy.

A missing Sitemap line is shown as information rather than an error because the directive is optional. A sitemap can also be submitted directly through search-engine tools or discovered by other supported methods.

A safe workflow after the check

1. Confirm the exact robots.txt URL, HTTP status, and response type.
2. Choose the crawler that matters to your policy instead of assuming one rule covers every provider use.
3. Test important public, private, search, product, and download paths separately.
4. Review the group used and the winning rule, not only the Allowed or Blocked badge.
5. Save the current live file before editing it.
6. Merge changes carefully, publish them at the correct host root, and repeat the live check.
7. Use server logs and provider verification methods when you need evidence of real crawler requests.

When the current file does not match your intended policy, use the generator for a reviewable draft, then merge it with any existing platform, search, and sitemap rules instead of replacing the live file blindly.