What Is robots.txt?

robots.txt is a plain-text crawl policy published at the root of a website origin. It tells compliant crawlers which URL paths they may or may not fetch, but it does not make content private and does not guarantee whether a URL will or will not be indexed.

Independent technical reference Updated Editorial approach Report a correction
Root-level crawler policy directing public and blocked website paths while private content uses separate server-side security

Copy-paste robots.txt example

User-agent: *
Allow: /
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

robots.txt is a crawl-control file. A site publishes it at the root of an origin, for example https://www.example.com/robots.txt, and compliant crawlers read it before requesting URLs.

Core directives

  • User-agent identifies the crawler or crawler group the following rules apply to.
  • Disallow defines URL paths the crawler should not fetch.
  • Allow can create a more specific allowed path within a broader restriction for crawlers that support it.
  • Sitemap is a widely supported extension used to advertise sitemap URLs. See Sitemap directive.
User-agent: *
Disallow: /private-area/
Allow: /private-area/public-info/

Sitemap: https://www.example.com/sitemap.xml

Five-step first-time robots.txt setup

  1. Open the exact origin’s /robots.txt, including the scheme and hostname used by visitors.
  2. Decide which crawler and path need restriction; do not block everything merely to hide administrative URLs.
  3. Start with a narrow Disallow. Add an Allow exception only where a child path must remain crawlable.
  4. Paste the proposed policy into the draft simulator and test both a blocked URL and a URL that must remain public.
  5. After publishing, check the live response. A successful draft result cannot verify the actual HTTP status or server override.

For example, with Disallow: /account/ and Allow: /account/help/, test /account/profile (blocked) and /account/help/guide (allowed). Neither result controls authentication or indexing. The Google robots.txt specification explains rule selection and HTTP error handling.

The file is scoped to an origin

A robots.txt file applies to the scheme, hostname and port where it is served. Rules from https://example.com/robots.txt do not automatically control a different subdomain such as https://shop.example.com/.

Crawling is not the same as indexing

Blocking a crawler from fetching a URL does not necessarily guarantee that the URL can never appear in a search index. A crawler can discover a URL from links or other sources without fetching its content. If your goal is index control, use the indexing mechanisms supported by the relevant search engine and make sure the crawler can actually access those signals when required.

robots.txt is not security

The file is public, and compliance is voluntary. Do not put secrets in robots.txt and do not use it to protect customer data, admin areas, private documents or paid content. Use authentication, authorization and server-side controls for that.

AI crawlers use the same file, but not the same purpose

Modern AI providers publish multiple user agents for training, search and user-triggered retrieval. The name matters. For example, OpenAI separates GPTBot from OAI-SearchBot. Build rules around the outcome you want rather than copying a blanket list.

Use the robots.txt Generator to build a policy and the Checker to test how a specific agent and path are interpreted.

FAQ

Where must robots.txt be placed?

It must be available at the lowercase top-level path /robots.txt for the exact host, such as https://example.com/robots.txt.

Is robots.txt a security feature?

No. It is a public instruction file for compliant crawlers. Protect private content with authentication, authorization, and server-side access checks.

Can robots.txt remove a page from search results?

Not reliably. It can stop crawling, but a search engine may still know the URL. Use noindex on an accessible page or a search-engine removal tool for index exclusion.

What happens when several rules match one URL?

The most specific matching path rule wins. If equally specific Allow and Disallow rules conflict, the Robots Exclusion Protocol prefers Allow.

Does the main domain’s robots.txt control subdomains?

No. The policy is scoped to the exact scheme, host, and port, so each subdomain needs its own robots.txt file.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Crawler identities matching specific robots.txt rule groups while unmatched agents follow a separate fallback group User-Agent in robots.txt: Definition and Matching

In robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.

 Website path tree with one directory disallowed while a more specific public path remains accessible Disallow in robots.txt: Meaning and Examples

Disallow tells the matching crawler not to request URL paths beginning with the specified pattern. Disallow: / blocks the entire host for that group, while an empty Disallow: rule is ignored. When rules conflict, the most specific matching path wins.

 Specific allowed corridor reaching one public resource inside a broadly restricted website area Allow in robots.txt: Rules, Priority, and Examples

To explicitly allow all compliant crawlers, use User-agent: * followed by Allow: /. That rule is usually unnecessary because crawling is allowed by default unless a matching Disallow rule blocks a path.

 Robots.txt generator controls and a generated rules document AI robots.txt Generator

Choose a preset or set individual crawler rules. The generator shows what will be allowed or blocked, highlights issues to review, and gives you a robots.txt file to publish yourself.

 Robots.txt checker inspecting a website and reporting allowed and blocked crawler access robots.txt Checker

Test the live robots.txt served by a website or paste unpublished rules to simulate a crawler across up to eight URL paths. Review the winning Allow or Disallow rule before deploying changes.