Add a standalone Sitemap line containing the full absolute URL of your XML sitemap or sitemap index, such as Sitemap: https://example.com/sitemap.xml. The line may appear anywhere in robots.txt and may be repeated, but it only helps discovery: it does not override Disallow rules or guarantee crawling or indexing.
User-agent: * Allow: / Sitemap: https://example.com/sitemap-index.xml Sitemap: https://example.com/news-sitemap.xml
To advertise an XML sitemap through robots.txt, add a standalone Sitemap: record with the complete URL, including the protocol and host:
Sitemap: https://example.com/sitemap.xmlDo not use a relative value such as Sitemap: /sitemap.xml. Search engines document the full absolute form, and it is clearer in multi-host or mixed-protocol setups.
Sitemap: is independent of User-agent, Allow, and Disallow. It can appear at the beginning or end of the file. Many sites place sitemap records at the bottom simply because they are easier to maintain there.
If you have several separate sitemap files, add more than one line. If you have a sitemap index, usually one line pointing to the index is enough:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap-index.xml
Sitemap: https://example.com/news-sitemap.xmlThe record tells a crawler where a sitemap is located. It does not override a matching robots.txt restriction. If /private/ is disallowed, listing a URL from that directory in the sitemap does not make the page crawlable for that agent.
The reverse is also true: removing a URL from a sitemap does not block it. Crawlers may still discover the URL through internal links, external links, feeds, browser data, or earlier crawl history. Use the appropriate Disallow, page-level indexing control, or server-side restriction for the outcome you actually need.
Open the sitemap URL before publishing the robots.txt line. It should return a successful response and valid sitemap XML rather than an HTML error page, redirect loop, login screen, or expired file. Common CMS locations include /sitemap.xml and /sitemap_index.xml, but the correct value is the address your site really serves.
A sitemap may be hosted separately in some verified cross-site configurations, but each sitemap still needs to describe the intended host correctly. Most ordinary sites should keep the setup simple and reference the sitemap generated for the same site.
Sitemap: line to the robots.txt file for the exact host.Disallow.For the record’s exact meaning, see the Sitemap directive glossary. For response failures, review robots.txt HTTP status codes.
It may appear anywhere because it is a standalone record rather than part of a User-agent group. Many sites place it at the end for readability.
Yes. You can list several absolute sitemap URLs. If a sitemap index already lists the individual files, one line pointing to that index is often simpler.
Yes. Use the complete URL including https:// and the host, for example https://example.com/sitemap.xml, rather than a relative path.
No. Sitemap helps URL discovery, while Allow and Disallow determine crawl permission for the matching crawler group. The access rule still applies.
No. It is a discovery hint. Search engines still evaluate crawl access, HTTP status, canonicalization, content quality, and their own indexing systems.
Sitemap Directive in robots.txt: Meaning and SyntaxThe Sitemap directive points crawlers to an XML sitemap from your robots.txt file. It helps discovery, but it does not allow blocked URLs, override Disallow rules, guarantee indexing, or replace sitemap submission tools.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
robots.txt HTTP Status Codes: 200, 3xx, 4xx, and 5xxA successful 2xx response lets crawlers parse robots.txt. Redirects may be followed, ordinary 4xx responses often mean no usable robots.txt restrictions, while 5xx and network failures are treated as temporary retrieval problems. Exact behavior and cache timing vary by crawler, so return a stable 200 response for a valid file and never use 401 or 403 as a substitute for crawl rules.
robots.txt Wildcards: * and $ Pattern MatchingIn robots.txt, * matches zero or more characters and $ anchors the match to the end of the URL pattern. These are limited pattern operators, not full regular expressions. Crawlers compare applicable Allow and Disallow paths and use the most specific match; if equally specific rules conflict, Allow should win.
Common robots.txt Mistakes and How to Fix ThemThe most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.