WordPress can serve a virtual robots.txt file even when no physical file exists. Add crawler-specific AI rules without blocking Googlebot, preserve the WordPress admin exception for admin-ajax.php, include the sitemap URL actually generated by your SEO setup, and verify the live root file.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap_index.xml
This example blocks several training-oriented crawlers while preserving ordinary public crawling and WordPress’s standard administration exception:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlReplace the sitemap address with the URL that actually exists on your site. WordPress core, an SEO plugin, or a custom sitemap system may use a different location such as /wp-sitemap.xml or /sitemap.xml.
WordPress core can generate robots.txt dynamically at the root URL. Its default public output includes a wildcard group that blocks /wp-admin/ while allowing /wp-admin/admin-ajax.php. The output can be changed through WordPress’s robots_txt filter, and many SEO plugins provide an editor that manages the same public response.
On common server configurations, a physical robots.txt file in the web root may be served before WordPress’s virtual response. Do not maintain competing versions without knowing which one the server returns. The only reliable source is the live file at https://your-domain.com/robots.txt.
robots_txt filter when rules should be versioned in code and survive theme changes.Avoid placing permanent robots.txt logic directly in a theme that may be replaced or updated. Also avoid copying a generic WordPress template over existing ecommerce, multilingual, membership, or staging rules without review.
Modern search engines need access to public CSS, JavaScript, images, and rendered pages. Broad blocks for /wp-content/, /wp-includes/, every query string, or every media file can interfere with rendering and diagnostics. Robots.txt is not a performance optimization for WordPress and does not secure plugin files, login endpoints, private media, or customer data.
WordPress can generate a virtual robots.txt response dynamically. Open /robots.txt on the live domain to see what crawlers actually receive.
WordPress’s default public output blocks /wp-admin/ and allows /wp-admin/admin-ajax.php. This is a crawling preference, not security for the administration area.
Use the URL that returns the live sitemap on your site. Common possibilities include /wp-sitemap.xml, /sitemap_index.xml, or /sitemap.xml, depending on the active setup.
Usually not. Search engines may need public CSS, JavaScript, images, and other assets to render pages correctly. Block only paths with a clear crawler-related reason.
A plugin, server cache, CDN cache, physical root file, or virtual WordPress output may still be serving another version. Inspect the live response and clear the relevant cache layer.
AI Robots.txt GeneratorChoose a policy mode, enter your website and sitemap, add any path or crawler overrides, then generate and download a robots.txt file. Publish it at the root of the correct host and verify the live rules with the checker before relying on them.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
Allow Search, Block AI Training robots.txt TemplateUse separate user-agent groups: block documented training or dataset agents such as GPTBot and ClaudeBot, while allowing ordinary search and dedicated AI-search crawlers.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
Common robots.txt Mistakes and How to Fix ThemThe most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.