robots.txt Templates: Copy-Paste Examples for Common Policies

Copy-paste starter robots.txt templates

If you need a robots.txt template quickly, start with the smallest policy that matches the result you actually want. These examples are intentionally short so you can see what each line changes before you merge it into the file your site already serves.

Allow normal crawling and declare the sitemap

This is a simple public-site starting point when you do not need crawler-specific restrictions:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Replace example.com with the canonical host. If the live file already contains platform rules, path restrictions, or additional sitemap declarations, preserve the rules you still need instead of replacing the whole file.

Keep ordinary crawling open and block selected training or dataset crawlers

This example keeps the wildcard group open while adding separate full-site blocks for three documented crawler names commonly associated with model training or dataset collection:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This does not mean that every AI-related product is blocked. Search crawlers, user-requested fetchers, provider-specific product controls, renamed agents, and clients that ignore robots.txt require separate decisions. Review the current provider documentation and use the narrower templates below when your policy needs more precision.

Choose a template by the result you need

A robots.txt template is useful when your goal is already clear: keep ordinary search available, restrict a documented training crawler, block a provider's dedicated agents, or adapt a platform-managed file. Start with the narrowest template that produces the result you actually want. A longer crawler list is not automatically safer.

Use a template for a known outcome and the Generator for a custom policy

Choose a template when one existing policy closely matches your goal and you are comfortable merging it into the file your site already serves. Use the robots.txt Generator when you need to combine several crawler decisions, add host-specific sitemap information, define blocked paths, or review a policy assembled from multiple choices.

A template and the Generator both produce a starting policy. Neither publishes to your hosting account or replaces the need to inspect the complete live file.

Build a custom robots.txt policy Check a live robots.txt file

Open the current live file before copying anything

Visit the exact host's public robots.txt URL first, such as https://example.com/robots.txt. Save the current text and determine what generates it. A CMS, ecommerce platform, SEO plugin, application route, CDN, or hosting layer may already control the response.

Do not replace the existing file blindly. It may contain platform rules, path restrictions, comments, crawler-specific groups, and one or more Sitemap: declarations that the chosen template does not include. The safest change is usually to merge only the groups required by the new policy.

• WordPress: confirm whether the public file is virtual, physical, filtered by a plugin, or served by the web server.
• Shopify: preserve the platform-generated output and use the supported robots.txt.liquid mechanism when customization is genuinely needed.
• Custom applications: check whether the route is generated dynamically and whether deployment or cache layers can overwrite a manual file.

Keep crawler purposes separate

One provider can publish different names for model-development crawling, search discovery, user-requested retrieval, or product-specific content-use controls. A provider name alone is therefore not a precise policy.

Before choosing a template, write down the outcome for each relevant category:

• Ordinary search: should Googlebot, Bingbot, Applebot, and other search crawlers remain able to fetch public pages?
• AI search discovery: do you want dedicated AI-search crawlers to discover and cite public content?
• Model training or product use: which documented training crawlers or product-control tokens do you intend to restrict?
• User-triggered retrieval: should an assistant be able to retrieve a public page after a person requests it?
• Dataset and SEO crawling: do Common Crawl, Ahrefs, Semrush, or other third-party crawlers fit your policy?

Use separate User-agent groups when the answers differ. Do not convert a narrow policy into User-agent: * unless blocking all compliant crawlers is intentional.

Check how new groups interact with existing path rules

A crawler-specific group does not automatically inherit restrictions from User-agent: *. Adding an exact group that contains only Allow: / can therefore open paths that were blocked only in the wildcard group.

User-agent: *
Disallow: /account/
Disallow: /internal-search/

User-agent: ExampleSearchBot
Allow: /

In this example, the specific group may be selected for ExampleSearchBot, while the wildcard restrictions are not merged into it. If that crawler must still avoid the two paths, repeat the applicable restrictions in its specific group and test the complete result.

This is why the narrowest useful template is usually safer than a file filled with explicit allow-all groups.

Move from a template to a verified live policy

1. Choose by outcome. Select the smallest template that matches the crawler purpose and visibility trade-off you intend.
2. Save the current live file. Keep a copy before editing so the platform rules and sitemap declarations can be restored.
3. Confirm every crawler name. Check the provider's current documentation before relying on an important block or allow decision.
4. Merge instead of replace. Add the required groups to the active source and preserve unrelated rules.
5. Review the complete policy. Look for duplicate groups, contradictory paths, exact groups that bypass wildcard restrictions, and an incorrect sitemap host.
6. Publish at the correct root. The policy must be served as /robots.txt for the exact scheme, host, and port it is intended to govern.
7. Verify the live response. Open the URL directly and confirm that the intended plain text is returned with a usable HTTP response.
8. Test crawler and path combinations. Use the Checker for the exact public and restricted URLs that matter.

Test more than the homepage

A homepage test cannot prove that the rest of the policy is correct. After publishing, test at least:

• the homepage with every crawler you deliberately blocked or allowed;
• a public article, product, or documentation URL;
• account, administration, internal-search, staging, and other restricted paths;
• a neighboring path that should produce the opposite result;
• ordinary search crawlers when search visibility must remain available;
• the canonical host and any separately served subdomain that has its own policy.

Inspect the selected group and winning rule, not only the final Allowed or Blocked label. Then review server logs separately when you need evidence of actual crawler requests.

Understand the limits of every robots.txt template

A template communicates crawling instructions to clients that choose to follow the protocol. It cannot authenticate a crawler, stop a spoofed user-agent, secure private content, delete material already collected elsewhere, or guarantee inclusion in search or AI answers.

Robots.txt also manages crawling rather than guaranteed removal from an index. Protect private, paid, personal, or administrative resources with authentication, authorization, application permissions, and server-side controls. Use indexing controls when the goal is to keep a page out of search results.

No template can remain complete forever. Provider names, product roles, platform behavior, and your own publishing goals can change. Review the live policy after major platform updates, domain migrations, new subdomains, crawler documentation changes, and unexpected traffic.

Need a policy that does not match one template?

Use the Generator to build crawler-specific groups and path rules, merge the result with the existing file, and then verify the published robots.txt policy. The service workflow is designed for cases where a copied example is not specific enough for the site you actually operate.

Protocol references

For the protocol's group and path-matching rules, review RFC 9309. For Google-specific crawling and indexing limitations, review Google Search Central's robots.txt introduction.