The most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / Sitemap: https://example.com/sitemap.xml
The file must be available as /robots.txt at the root of the exact host, protocol, and port it controls. A file at /folder/robots.txt is not the policy for the site root. Likewise, the file on www.example.com does not automatically control shop.example.com.
This two-line rule requests a complete crawl shutdown for every compliant agent:
User-agent: *
Disallow: /It can affect Googlebot, Bingbot, AI search crawlers, monitoring tools, and other services. Use exact user-agent groups when your goal is selective AI training control.
Robots.txt controls requests, not guaranteed removal from search results. A blocked URL can still be discovered from links and shown without a normal snippet. If a page must not appear in search, use a crawlable noindex directive, authentication, or the search engine’s removal process as appropriate.
The Robots Exclusion Protocol is not access authorization. The file is public, exposes the paths it names, and can be ignored by a malicious client. Never use it to protect private dashboards, customer files, licensed downloads, or confidential directories.
Rule paths should begin with / and are evaluated against the URL path. They are case-sensitive for crawlers that follow the standard. A rule for /Private/ may not match /private/. Test representative URLs rather than assuming a directory name is correct.
For matching rules, compliant crawlers use the most specific path. When an Allow and Disallow rule have equally specific matching paths, Allow wins under RFC 9309. File order is not a safe substitute for understanding specificity.
User-agent: *
Disallow: /account/
Allow: /account/help/Repeated groups for the same user-agent may be combined by compliant parsers. A missing colon, accidental HTML response, invalid encoding, or copied smart punctuation can cause rules to be ignored. Serve plain UTF-8 text and inspect the public response, not only the editor.
Disallowing essential CSS, JavaScript, or image assets can stop a search crawler from understanding the page as users see it. Restrict unimportant resources only after confirming that rendering and indexing are not affected.
Training, search, product-control, and user-triggered agents are often independent. Blocking GPTBot does not automatically block OAI-SearchBot; blocking Google-Extended does not block Googlebot. Verify the exact published token and purpose.
Crawlers may cache robots.txt, and changes can take time to propagate. A CDN, reverse proxy, or CMS can also serve an old file. Confirm the live response from multiple locations and allow the provider’s documented refresh window.
Any client can claim to be Googlebot or another crawler. Use the provider’s documented DNS or published-IP verification method before allowing, blocking, or rate-limiting traffic based on identity.
Use the robots.txt checker with real allowed and blocked URLs. Review wildcard groups, duplicate agents, empty rules, sitemap lines, response status, and the effect on normal search crawlers before publishing.
No, it is valid when you intentionally want every compliant crawler to avoid the entire host. It is a mistake when the real goal is only to block selected AI or dataset crawlers.
Robots.txt can stop crawling while the URL is still discovered through links. Search engines may display the address without a normal snippet. Use an appropriate indexing or access-control method for removal.
No. Matching is based on path specificity. Under RFC 9309, the most specific rule wins, and Allow wins when equally specific Allow and Disallow rules match.
No. It is a public advisory file and not authorization. Protect private resources with authentication and server-side access controls.
Do not trust the user-agent string alone. Use the provider’s official DNS verification process or published IP ranges, depending on its documentation.
robots.txt Definition, Syntax, and ExamplesRobots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.
User-Agent in robots.txt: Definition and MatchingIn robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.
Disallow in robots.txt: Meaning and ExamplesDisallow tells the matching crawler not to request URL paths beginning with the specified pattern. Disallow: / blocks the entire host for that group, while an empty Disallow: rule is ignored. When rules conflict, the most specific matching path wins.
Allow in robots.txt: Meaning and PriorityAllow explicitly permits the matching crawler to request a path. It is most useful as a narrow exception inside a broader Disallow rule. The longest matching path wins; if equally specific Allow and Disallow rules conflict, Allow is preferred.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.