Robots.txt and AI Crawler Glossary

Use this glossary for concise definitions of crawler roles, robots.txt directives, and newer AI-content control terms. Each entry explains what the term means, what it does not mean, and where it fits in a practical website policy.

Use precise terms before writing crawler rules

Many robots.txt mistakes begin with category confusion. A crawler can support ordinary search, AI search, model development, a user-requested fetch, an open dataset, or an SEO tool. Likewise, a robots.txt directive can control crawling without removing a URL from an index or securing the underlying content.

What does robots.txt mean?

robots.txt is a plain-text file at the root of a website origin that asks compliant automated clients which URL paths they may fetch. It does not password-protect pages, request Google indexing, or guarantee removal of an indexed URL.

User-agent: *
Disallow: /private/
Allow: /private/public/

This sample restricts requests under /private/ but permits the more specific /private/public/ path for crawlers that implement standard matching. For the syntax of each line, consult User-agent, Disallow, and Allow. For a path-by-path simulation, use the draft tester.

Core robots.txt concepts

Start with robots.txt and User-Agent, then review Disallow and Allow. These entries explain how groups target crawler tokens and how path-specific rules are interpreted. The Sitemap directive covers discovery of XML sitemaps, while Crawl-delay explains a non-standard extension that is not supported uniformly.

AI crawler categories

The AI crawler entry introduces the umbrella term. Use the separate definitions for AI training crawler and AI search crawler when deciding whether training preferences and search visibility should differ. Provider-specific user-triggered agents may form another category because they fetch after a person asks for a page rather than crawling automatically.

Signals beyond basic crawl access

The Content Signal entry explains emerging mechanisms intended to communicate preferences about downstream content use. Such signals should not be confused with authentication, copyright licensing, or a guaranteed technical block.

Move from a definition to a tested policy

After reading a term, open the linked guide or crawler page for examples and consequences. Use the generator to assemble a policy and the checker to test the live file. Definitions describe the mechanism; only the complete published robots.txt and the target crawler’s behavior determine the practical result.

Browse this section

Public website connected to separate AI crawler categories for model development, search discovery, and user-requested retrieval

AI Crawler: Definition, Types, and robots.txt Control

An AI crawler is a web agent used by an AI provider to collect, discover, index, or retrieve online content. The label is broad: training crawlers, AI search crawlers, and user-requested fetchers can have different purposes and separate robots.txt tokens.

 AI search crawler discovering public webpages and connecting selected sources to a grounded answer with citations

AI Search Crawler: Definition, Visibility, and Control

An AI search crawler discovers and analyzes public web pages for search results, grounded answers, snippets, citations, or source links. Allowing it can support visibility in that AI search product; blocking it may reduce discovery, but neither choice guarantees inclusion or exclusion from every AI surface.

 Robots.txt control stopping public web content from entering an AI model-training pipeline while search discovery remains separate

AI Training Crawler: Definition and robots.txt Control

An AI training crawler collects public web content that may contribute to model development, improvement, evaluation, or safety work. A crawler-specific Disallow rule can signal that future collection is not permitted, but it does not erase previously acquired data or secure private content.

 Specific allowed corridor reaching one public resource inside a broadly restricted website area

Allow in robots.txt: Rules, Priority, and Examples

To explicitly allow all compliant crawlers, use User-agent: * followed by Allow: /. That rule is usually unnecessary because crawling is allowed by default unless a matching Disallow rule blocks a path.

 Content accessed from a website and then governed by separate search, AI-input, training, and reuse preference paths

Content-Signal in robots.txt: Meaning and Examples

Content-Signal is a non-standard robots.txt directive for expressing how accessed content may be used, including search indexing, real-time AI input, model training, and newer reuse levels. It states a preference or rights reservation; it does not itself block requests or guarantee that every crawler will comply.

 Crawler requests reaching a server at measured intervals while a separate lane shows that Crawl-delay support varies

Crawl-delay in robots.txt: Meaning and Crawler Support

Crawl-delay is a non-standard robots.txt extension. Google documents it as unsupported, while Bingbot documents support with crawler-specific timing semantics. Use it only when logs show a real crawl-load problem and the crawler\u2019s current documentation confirms how the value is interpreted.

 Website path tree with one directory disallowed while a more specific public path remains accessible

Disallow in robots.txt: Meaning and Examples

Disallow tells the matching crawler not to request URL paths beginning with the specified pattern. Disallow: / blocks the entire host for that group, while an empty Disallow: rule is ignored. When rules conflict, the most specific matching path wins.

 Robots.txt pointing crawlers to a sitemap while crawl access rules remain a separate control layer

Sitemap Directive in robots.txt: Definition and Syntax

The Sitemap directive is a widely supported robots.txt extension that tells crawlers where a site’s XML sitemap or sitemap index can be found. Use an absolute URL. It advertises the sitemap location but does not force crawling or indexing.

 Crawler identities matching specific robots.txt rule groups while unmatched agents follow a separate fallback group

User-Agent in robots.txt: Definition and Matching

In robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.

 Root-level crawler policy directing public and blocked website paths while private content uses separate server-side security

What Is robots.txt? Scope, Rules and Limitations

robots.txt is a plain-text crawl policy published at the root of a website origin. It tells compliant crawlers which URL paths they may or may not fetch, but it does not make content private and does not guarantee whether a URL will or will not be indexed.

FAQ

What is the difference between crawling and indexing?

Crawling is requesting and processing a resource. Indexing is storing or using information so it can appear in a search system. Blocking crawl access does not always guarantee removal of a known URL.

Is an AI crawler always used for model training?

No. AI-related agents may support model development, search discovery, or a user-initiated request. Use the provider’s documented token and purpose.

Are Allow and Disallow security permissions?

No. They are robots.txt crawling directives for compliant automated clients. They do not authenticate visitors or protect private content.

Is Crawl-delay part of the core robots.txt standard?

No. It is a non-standard extension supported by some crawlers and ignored by others. Check the target provider’s documentation.

Why does the glossary link to guides and tools?

A definition explains a term, while the guides show policy trade-offs and the tools help generate or verify the complete live robots.txt file.