Content-Signal in robots.txt: Meaning and Examples

What Content-Signal is

Content-Signal is an extension used to publish machine-readable preferences about what an automated system may do with content after accessing it. Cloudflare introduced the Content Signals Policy to address a limitation of ordinary robots.txt: Allow and Disallow describe crawler access, but they do not clearly describe permitted downstream uses.

The directive is not part of the core Robots Exclusion Protocol in RFC 9309. Recognition and enforcement depend on the crawler or service. It should therefore be treated as a policy and rights signal, not as a universal technical control.

The original three purpose signals

• search — building a search index and returning links or short excerpts. In the Cloudflare policy, this category does not include AI-generated search summaries.
• ai-input — using content as real-time input to an AI model, such as retrieval-augmented generation, grounding, or an AI-generated answer.
• ai-train — training or fine-tuning AI models.

Each purpose uses yes or no. Omitting a purpose does not mean yes or no; it means that no preference for that purpose is expressed through this directive.

Basic syntax

A publisher that wants ordinary search indexing, but does not want model training or real-time AI-answer input, could publish:

User-agent: *
Content-Signal: search=yes, ai-train=no, ai-input=no
Allow: /

The access rule and use preference are separate. Allow: / says that compliant crawlers may fetch the public content. The Content-Signal line states the publisher’s requested limits on how the fetched content may be used.

Newer content-use values

Cloudflare’s current managed robots.txt documentation also describes a use preference with three levels:

• use=immediate — interact with the content without storing and reusing it.
• use=reference — index, excerpt, and link back to the source.
• use=full — summarize and reproduce the content.

This extension is evolving. Check current documentation and crawler support before relying on a newly introduced value. Do not assume that older parsers, search engines, or AI operators understand every field.

Content-Signal versus Allow and Disallow

Practical implementation

1. Decide separately whether you allow search indexing, AI-answer input, training, and broader reuse.
2. Add the Content Signals Policy comments if you want to state the definitions and legal reservation supplied by the policy project.
3. Add one machine-readable Content-Signal line to the relevant User-agent group.
4. Keep crawler-specific blocks where you need access control for known bots.
5. Use the robots.txt checker to confirm that the file remains syntactically clear, then review real logs and provider documentation.

Content signals complement the distinctions explained in AI Training Crawler and AI Search Crawler. They do not replace an enforceable licensing or access-control strategy.