A practical small-business robots.txt file should keep ordinary search discovery open, block only the crawler purposes you have intentionally rejected, and point compliant crawlers to your sitemap. Review every user-agent before publishing because robots.txt is a public policy file, not a security control.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
This example is designed for a typical public small-business website that wants to remain discoverable in conventional and AI-assisted search while opting out of selected training or dataset uses. Replace the sitemap URL before publishing:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThe specific groups express a narrow policy. The wildcard group keeps other compliant crawlers open, including ordinary search crawlers unless another rule elsewhere in the file applies to them.
GPTBot and ClaudeBot are documented training-related crawlers.CCBot collects pages for Common Crawl’s open web dataset.Google-Extended and Applebot-Extended are product-control tokens rather than separate page-crawling bots. Blocking them can restrict specified generative-AI uses while leaving the associated search crawler available.User-agent: * with Allow: / leaves the rest of the public site open to compliant crawlers.This is a starting policy, not a universal answer. A local service business, an online store, and a subscription publisher may reasonably choose different rules.
Do not add OAI-SearchBot, Claude-SearchBot, or PerplexityBot to the block unless you also want to reduce discovery in those providers’ search experiences. User-triggered agents such as ChatGPT-User, Claude-User, and Perplexity-User are another category again; provider behavior differs, and robots.txt may not be sufficient for requests initiated by a person.
https://your-domain.example/robots.txt.Robots.txt does not hide URLs, authenticate visitors, or stop clients that ignore the protocol. Protect private areas with login, authorization, and server-side controls.
The template does not block Googlebot or Bingbot. However, merge it carefully with your existing file because a separate wildcard or crawler-specific rule can still affect search crawling.
No. This is a policy choice. Consider discovery goals, content value, licensing concerns, server load, and whether you want your public pages available for particular AI uses.
Yes. Each specific user-agent group is independent. Keep only the groups that match your policy and remove rules you do not intend to publish.
No. Crawler-specific groups normally take precedence for their matching user agent, and path matching rules still matter. Test the complete live file rather than reading one line in isolation.
No. Robots.txt is public and voluntary. Use authentication, authorization, and server-side access controls for confidential or restricted resources.
AI Robots.txt GeneratorChoose a policy mode, enter your website and sitemap, add any path or crawler overrides, then generate and download a robots.txt file. Publish it at the root of the correct host and verify the live rules with the checker before relying on them.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
Allow Search, Block AI Training robots.txt TemplateUse separate user-agent groups: block documented training or dataset agents such as GPTBot and ClaudeBot, while allowing ordinary search and dedicated AI-search crawlers.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
How to Check Whether AI Crawlers Are BlockedFetch the live robots.txt file, select the crawler’s exact product token, and test the specific URL against the matching group and longest Allow or Disallow rule. Then check HTTP responses, caching, firewall rules, and logs, because a crawler may be allowed by robots.txt yet blocked by the server—or disallowed by robots.txt while an unidentified scraper still reaches the page.