Generate robots.txt Rules for AI Crawlers

What this generator creates—and what it does not

The generator creates a plain-text robots.txt draft for the exact website host you enter. It can add crawler-specific User-agent groups, wildcard path rules, a sitemap declaration, and an optional Content Signals preference. The result remains visible before download so you can review every explicit allow, block, path, and warning.

The tool does not connect to your hosting account, modify an existing file, authenticate a crawler, or guarantee that an automated client will comply. Publication remains under your control. If the site already has a robots.txt file, merge the draft into the current policy instead of replacing the file without review.

Choose a preset by its exact outcome

Preset names are shortcuts. Check the exact crawler names shown in the result before publishing, because providers may use separate names for training, search, and user-requested access.

• Balanced — keep search, block common training: blocks common AI-training, product-control, and dataset crawlers while leaving ordinary search and AI-search discovery available.
• Block listed AI crawlers: blocks the AI-training, AI-search, user-triggered, product-control, and dataset crawler names available in the generator. New, unidentified, or non-compliant clients may still require other controls.
• Explicitly allow AI search, block training: creates exact allow groups for listed AI-search crawlers and exact block groups for common training-related crawlers.
• Block user-triggered AI agents: targets listed agents that fetch pages after a person asks an AI product to open or use a page.
• Custom: starts without crawler-specific rules so you can choose individual crawler rules yourself.

Use advanced overrides only when you can explain why a specific crawler needs different treatment from the selected preset. Provider documentation and crawler behavior can change, so check the linked official source before publishing a high-impact restriction.

Understand exact groups and the wildcard group

Robots.txt processors choose the most specific matching User-agent group. An exact crawler group does not inherit rules from User-agent: *. This matters when you combine explicit crawler permissions with wildcard blocked paths.

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /members/

In this example, the exact OAI-SearchBot group allows the whole site. The wildcard /members/ restriction is not automatically added to that exact group. If an explicitly allowed crawler must also avoid a path, edit its exact group accordingly and verify the result. The generator highlights this situation when your settings create both exact allows and wildcard blocked paths.

Wildcard path rules are public crawling instructions. They are not a safe way to hide account pages, customer information, paid resources, private files, or administration areas. Protect sensitive content with authentication, authorization, server configuration, and application-level access checks.

Use the policy review before copying the file

The generated result includes more than the text file. The policy review identifies:

• the selected policy and its exact output;
• every crawler name explicitly blocked;
• every crawler name explicitly allowed;
• the paths placed under the wildcard group;
• whether Content Signals were included;
• warnings that require manual review before publication.

Pay particular attention to warnings about ordinary search crawlers, exact allow groups, and private paths. A syntactically valid file can still express the wrong visibility policy for your site.

Merge the draft with an existing robots.txt file

Do not assume a generated draft should replace the current file. Existing robots.txt content may contain platform rules, search-engine directives, crawl controls for third-party tools, multiple sitemap declarations, comments maintained by another system, or paths required by an ecommerce or content-management platform.

1. Open the current live /robots.txt file and save a copy.
2. Compare its crawler groups, paths, and sitemap lines with the generated draft.
3. Keep every existing rule that still serves a documented purpose.
4. Avoid creating duplicate groups for the same crawler unless you understand how the groups will be combined.
5. Publish the merged file only after reviewing the complete result as one policy.

If your platform generates robots.txt dynamically, use the platform's supported editing method rather than uploading a static file that may be ignored or overwritten.

What the optional Content Signals line means

When enabled, the generator places the following preference inside the wildcard group:

User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no

This expresses a preference to allow search use while declining AI input and AI training uses. Support varies by crawler and validator, and the directive does not replace crawler-specific rules. It also does not provide authentication, authorization, copyright enforcement, or a technical barrier against non-compliant clients.

Content Signals must appear inside a crawler group, so the generator places the line immediately after User-agent: *. After publishing, verify the live file with the checker and remember that support in third-party validators may vary.

Publish at the correct host root

Publish the final file at the root of the exact scheme and host it should govern, for example https://example.com/robots.txt. A file for www.example.com does not automatically govern shop.example.com, and a file placed in a subdirectory does not control the entire host.

1. Open the published URL directly and confirm that it returns the intended plain text.
2. Check the HTTP status and make sure a login page, redirect loop, error document, or HTML template was not returned instead.
3. Use the live robots.txt checker with important crawler names and paths.
4. Confirm the winning group and rule for each test, especially when exact and wildcard groups coexist.
5. Review server logs separately when you need evidence of real requests. A claimed user-agent string alone does not authenticate its sender.

When the generated draft needs additional review

Do not publish the draft unchanged when the site uses a platform-managed robots.txt file, several hosts or subdomains, a reverse proxy or CDN that rewrites the response, sensitive routes, complex wildcard patterns, legal or licensing obligations, or existing rules you do not understand. In those cases, treat the generator as a structured starting point and verify the final policy with the responsible developer, platform documentation, or legal adviser where appropriate.