Build and Verify robots.txt Rules for AI Crawlers

Build a crawler policy around the result you want

A robots.txt file is most useful when each rule supports a clear publishing decision. “Block AI” is not one decision: a provider may use separate tokens for model training, AI search, user-requested page access, ordinary search indexing, or product-level controls. Blocking an entire provider without checking those roles can remove access that you intended to keep.

For example, OpenAI documents GPTBot, OAI-SearchBot, and ChatGPT-User as separate tokens. A publisher can therefore express a selective policy instead of treating training, search discovery, and user-triggered retrieval as the same activity:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

This example is a starting point, not a universal recommendation. Your policy may depend on search visibility, licensing, server load, customer expectations, and whether an assistant should be able to fetch a public page after a user requests it. Review the current provider documentation before relying on any token.

Publish and verify with a repeatable workflow

1. Define the intended outcome. Decide whether you want to preserve conventional search, allow AI search discovery, permit user-triggered access, restrict training use, reduce dataset collection, or limit third-party SEO crawling.
2. Generate crawler-specific groups. Use the generator to create separate User-agent groups instead of reaching immediately for User-agent: *.
3. Review the complete file. Keep any existing rules, sitemap declarations, host-specific paths, and platform requirements that still apply. A generated file should not erase a working policy by accident.
4. Publish at the root of the correct host. The file normally belongs at a URL such as https://example.com/robots.txt. A file placed inside a subdirectory does not govern the entire origin.
5. Test the live response. Use the checker after deployment, select important crawlers and paths, and confirm the HTTP response, applicable group, and winning rule.
6. Monitor real requests. Server logs can show which user-agents request the site, but a user-agent string alone does not prove that the request came from the organization named in that string.

What the live checker can confirm

The checker requests the public robots.txt file for the submitted host and evaluates a selected crawler and path. It can help identify:

• whether the file is reachable and which HTTP status was returned;
• which User-agent group applies to the selected crawler;
• which matching Allow or Disallow rule is the most specific;
• whether wildcard patterns and end anchors affect the tested path;
• which sitemap declarations are present in the fetched file.

The checker cannot guarantee that every automated client will comply, authenticate a crawler from its user-agent string, or determine the legal meaning of a provider’s use of public content. Those questions require server-side verification, provider documentation, contractual review, or real access controls.

Use robots.txt as a policy signal, not a security boundary

Robots.txt is publicly readable and relies on voluntary compliance. It does not hide a URL, require a password, encrypt a file, prevent copying by a non-compliant scraper, or guarantee removal from search results. Never publish secrets, private customer data, paid content, administration paths, or confidential files and assume that a Disallow rule makes them safe.

Protect sensitive resources with authentication, authorization, server configuration, rate limits, network controls, and application-level checks. Use indexing controls supported by the relevant search engine when the goal is to prevent a page from appearing in search rather than merely controlling crawling.

Check the exact crawler name before you publish

Use the crawler directory to find the exact User-agent name, understand what the crawler is used for, and open the provider's official documentation. Use a template when your goal matches a common policy, or read the guides when you need to compare options and understand their consequences.

Crawler names and provider policies can change. Before publishing an important allow or block rule, confirm the current provider documentation, test the live file, and update the policy when a provider introduces or changes a crawler. Report a broken source or inaccurate description through the contact page.