Generate a robots.txt Policy for AI Crawlers

What the generator creates

This tool builds a plain-text robots.txt policy for the host you enter. The result can include crawler-specific User-agent groups, site-wide or path-specific Allow and Disallow rules, and a Sitemap directive.

The generator does not change your website automatically. It gives you a file to review, download, publish at the root of the correct host, and verify after deployment.

Choose a policy that matches your goal

Start with the outcome you want rather than blocking every automated client with one broad rule.

• Balanced: restrict common AI training and dataset crawlers while keeping regular search and supported AI-search crawlers available.
• Block all AI crawlers: apply a stricter policy to known AI training, AI-search, and user-triggered agents without automatically blocking ordinary Googlebot or Bingbot.
• Allow AI search, block training: keep supported AI-search crawlers open while restricting common training crawlers.
• Block user-triggered AI agents: restrict agents that fetch pages after a user request without automatically blocking training or search crawlers.
• Custom: start without a preset and set the required crawler rules individually.

Advanced custom bot rules override the selected preset for the individual User-agent token. Use them only when you understand why that crawler needs a different rule.

Add paths and sitemap information carefully

Custom blocked paths are written under User-agent: *, so they may affect every crawler that relies on the wildcard group. Add only paths you genuinely want compliant crawlers to avoid. Do not list private URLs as a substitute for authentication, because robots.txt is publicly accessible.

If you leave the Sitemap field empty, the generator uses the usual /sitemap.xml location for the entered host. Sites using WordPress or an SEO plugin may instead publish a sitemap index such as /sitemap_index.xml. Enter the address that actually exists on your site.

Review experimental Content Signals separately

The optional Content Signals block expresses preferences for search use, AI input, and AI training. These signals are experimental, are not part of the core access-control rules, and may not be supported by every crawler. Keep crawler-specific User-agent rules when you need an explicit robots.txt instruction.

Publish and verify the result

1. Review the generated groups and confirm that regular search crawlers are not blocked unintentionally.
2. Download the result as robots.txt.
3. Merge it carefully with any rules already used by your website instead of overwriting a working file blindly.
4. Publish it at the root of the relevant host, for example https://example.com/robots.txt.
5. Open the published file in a browser and use the robots.txt checker to test important crawlers and paths.

Remember that robots.txt controls crawling instructions for clients that choose to follow the protocol. It is not authentication, authorization, or a firewall.