How to Add a Sitemap to robots.txt Correctly

Add a standalone Sitemap line containing the full absolute URL of your XML sitemap or sitemap index, such as Sitemap: https://example.com/sitemap.xml. The line may appear anywhere in robots.txt and may be repeated, but it only helps discovery: it does not override Disallow rules or guarantee crawling or indexing.

Independent technical reference Updated Editorial approach Report a correction
Robots.txt pointing crawlers to a sitemap index while crawl permissions remain independent

Copy-paste robots.txt example

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap-index.xml
Sitemap: https://example.com/news-sitemap.xml

Use a full Sitemap URL

To advertise an XML sitemap through robots.txt, add a standalone Sitemap: record with the complete URL, including the protocol and host:

Sitemap: https://example.com/sitemap.xml

Do not use a relative value such as Sitemap: /sitemap.xml. Search engines document the full absolute form, and it is clearer in multi-host or mixed-protocol setups.

The line is not part of a crawler group

Sitemap: is independent of User-agent, Allow, and Disallow. It can appear at the beginning or end of the file. Many sites place sitemap records at the bottom simply because they are easier to maintain there.

If you have several separate sitemap files, add more than one line. If you have a sitemap index, usually one line pointing to the index is enough:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap-index.xml
Sitemap: https://example.com/news-sitemap.xml

A sitemap does not grant crawl access

The record tells a crawler where a sitemap is located. It does not override a matching robots.txt restriction. If /private/ is disallowed, listing a URL from that directory in the sitemap does not make the page crawlable for that agent.

The reverse is also true: removing a URL from a sitemap does not block it. Crawlers may still discover the URL through internal links, external links, feeds, browser data, or earlier crawl history. Use the appropriate Disallow, page-level indexing control, or server-side restriction for the outcome you actually need.

Choose the correct sitemap address

Open the sitemap URL before publishing the robots.txt line. It should return a successful response and valid sitemap XML rather than an HTML error page, redirect loop, login screen, or expired file. Common CMS locations include /sitemap.xml and /sitemap_index.xml, but the correct value is the address your site really serves.

A sitemap may be hosted separately in some verified cross-site configurations, but each sitemap still needs to describe the intended host correctly. Most ordinary sites should keep the setup simple and reference the sitemap generated for the same site.

Publishing workflow

  1. Open the sitemap and confirm its final URL and HTTP status.
  2. Add the absolute Sitemap: line to the robots.txt file for the exact host.
  3. Publish robots.txt at the host root.
  4. Open the live file and confirm that the line was not altered by a CMS, CDN, or hosting layer.
  5. Use the robots.txt checker to verify syntax and any conflicting access rules.
  6. Optionally submit the sitemap in search-engine webmaster tools for processing reports.

Common mistakes

  • Using a relative sitemap path.
  • Pointing to an HTML page instead of XML.
  • Referencing a staging, HTTP, or obsolete host.
  • Assuming the sitemap overrides Disallow.
  • Expecting submission to guarantee indexing.

For the record’s exact meaning, see the Sitemap directive glossary. For response failures, review robots.txt HTTP status codes.

FAQ

Where should the Sitemap line go in robots.txt?

It may appear anywhere because it is a standalone record rather than part of a User-agent group. Many sites place it at the end for readability.

Can robots.txt contain multiple Sitemap lines?

Yes. You can list several absolute sitemap URLs. If a sitemap index already lists the individual files, one line pointing to that index is often simpler.

Should the Sitemap value be an absolute URL?

Yes. Use the complete URL including https:// and the host, for example https://example.com/sitemap.xml, rather than a relative path.

Does a Sitemap line override Disallow?

No. Sitemap helps URL discovery, while Allow and Disallow determine crawl permission for the matching crawler group. The access rule still applies.

Does adding a sitemap guarantee indexing?

No. It is a discovery hint. Search engines still evaluate crawl access, HTTP status, canonicalization, content quality, and their own indexing systems.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Robots.txt pointing crawlers to a sitemap while crawl access rules remain a separate control layer Sitemap Directive in robots.txt: Meaning and Syntax

The Sitemap directive points crawlers to an XML sitemap from your robots.txt file. It helps discovery, but it does not allow blocked URLs, override Disallow rules, guarantee indexing, or replace sitemap submission tools.

 Robots.txt checker inspecting a website and reporting allowed and blocked crawler access robots.txt Checker

Enter the exact website address, choose a crawler, and test a path. The checker reads the live robots.txt file and explains the group and rule that determine access.

 Crawler receiving successful, redirected, missing, and temporary-failure responses for robots.txt robots.txt HTTP Status Codes: 200, 3xx, 4xx, and 5xx

A successful 2xx response lets crawlers parse robots.txt. Redirects may be followed, ordinary 4xx responses often mean no usable robots.txt restrictions, while 5xx and network failures are treated as temporary retrieval problems. Exact behavior and cache timing vary by crawler, so return a stable 200 response for a valid file and never use 401 or 403 as a substitute for crawl rules.

 URL paths passing through wildcard, exact-ending, and longest-match robots.txt rule selection robots.txt Wildcards: * and $ Pattern Matching

In robots.txt path rules, * matches zero or more characters and $ anchors the match to the end. The most specific applicable rule decides the result, not the last line in the file. Build the complete policy, then test exact crawler and URL combinations before publishing.

 Robots.txt workspace showing wrong location, overbroad blocking, path conflicts, and false security assumptions Common robots.txt Mistakes and How to Fix Them

The most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.