The Sitemap directive points crawlers to an XML sitemap from your robots.txt file. It helps discovery, but it does not allow blocked URLs, override Disallow rules, guarantee indexing, or replace sitemap submission tools.
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/news-sitemap.xml
The Sitemap: directive is a widely supported robots.txt extension that tells a crawler where to find an XML sitemap or sitemap index. It is a discovery hint: the crawler can fetch the referenced file, inspect the listed URLs, and decide which pages to crawl according to its own scheduling and your access rules.
A Sitemap line is not part of a User-agent group. It applies as a standalone record, so it does not need an Allow or Disallow line above it. Google documents that the line can appear anywhere in robots.txt; many site owners place sitemap entries at the end because they are easier to find and maintain there.
Use the full absolute URL, including the protocol and host:
Sitemap: https://example.com/sitemap.xmlThe value should resolve to a valid sitemap or sitemap index. Relative values such as /sitemap.xml are a common mistake. If your site has several valid sitemap files, add one Sitemap: line for each, or reference a sitemap index that lists them.
A sitemap does not grant crawl access. If /private/ is disallowed, listing a URL from that directory in a sitemap does not override the robots.txt rule. The opposite is also true: removing a URL from a sitemap does not prevent crawlers from discovering it through internal links, external links, feeds, or earlier crawl data.
Sitemaps also do not guarantee crawling, indexing, ranking, or appearance in an AI-generated answer. They are structured discovery signals. Search engines still evaluate accessibility, canonicalization, response status, content quality, and their own crawl priorities.
The robots.txt file must be served from the root of the exact host it controls, but the sitemap URL itself may use a dedicated sitemap path or a sitemap index. Keep the referenced file publicly reachable and return a successful HTTP response. For multi-host setups, verify that every sitemap contains URLs it is authorized to describe and that each host publishes the correct robots.txt policy.
Adding the directive is useful because compliant crawlers can discover the sitemap whenever they fetch robots.txt. You can also submit sitemaps directly in Google Search Console or Bing Webmaster Tools to receive processing information and error reports. These methods complement each other; using one does not make the other harmful or redundant.
Disallow.https://your-domain.com/robots.txt and confirm the exact sitemap URL.No. Sitemap is a standalone record, not an Allow or Disallow rule for one crawler group. It is commonly placed at the end of robots.txt for readability.
Yes. You may list multiple absolute sitemap URLs, although a sitemap index is often easier to manage when a site has many sitemap files.
No. A crawler that honors robots.txt should still respect the matching Disallow rule even when the blocked URL appears in a sitemap.
No. It helps crawlers discover URLs, but crawling and indexing remain subject to access, canonicalization, response quality, content quality, and each engine’s own priorities.
You can. Direct submission in Google Search Console or Bing Webmaster Tools provides reporting, while the robots.txt directive offers automatic discovery. The methods can be used together.
How to Add a Sitemap to robots.txt CorrectlyAdd a standalone Sitemap line containing the full absolute URL of your XML sitemap or sitemap index, such as Sitemap: https://example.com/sitemap.xml. The line may appear anywhere in robots.txt and may be repeated, but it only helps discovery: it does not override Disallow rules or guarantee crawling or indexing.
robots.txt Definition, Syntax, and ExamplesRobots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
robots.txt HTTP Status Codes: 200, 3xx, 4xx, and 5xxA successful 2xx response lets crawlers parse robots.txt. Redirects may be followed, ordinary 4xx responses often mean no usable robots.txt restrictions, while 5xx and network failures are treated as temporary retrieval problems. Exact behavior and cache timing vary by crawler, so return a stable 200 response for a valid file and never use 401 or 403 as a substitute for crawl rules.
Common robots.txt Mistakes and How to Fix ThemThe most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.