robots.txt for AI Search Visibility: Keep Discovery Open

To support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.

Independent technical reference Updated Editorial approach Report a correction
Public pages moving through AI search discovery and technical access toward possible source citation

Copy-paste robots.txt example

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

AI search visibility starts with the correct crawler

AI answer engines do not all use one universal crawler. OpenAI documents OAI-SearchBot for ChatGPT search, Anthropic documents Claude-SearchBot for search experiences, and Perplexity documents PerplexityBot for surfacing and linking websites in its results. These agents are separate from training-oriented and user-triggered agents.

If you want your public pages to be eligible for discovery, address the exact search token rather than allowing every crawler from the provider.

Publish explicit allow groups

A visibility-oriented file can make search access clear:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Explicit Allow: / groups are useful when a broader wildcard group blocks other agents. Review the whole file because a crawler selects the most specific matching user-agent group and then evaluates the applicable path rules. A rule that looks correct in isolation may still be defeated by a typo, duplicate group, or more specific path restriction.

Separate search from training preferences

You can allow AI search discovery while restricting model-development agents:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

This does not create a universal legal or technical policy, but it records a clear provider-specific preference. Use the selective AI policy guide for a larger multi-provider example.

Robots.txt is only one accessibility layer

An allowed crawler can still fail to reach a page because of a login, WAF challenge, blocked IP range, repeated server errors, redirect loop, or inaccessible rendering resources. Confirm that the page returns a stable successful response and that legitimate crawler requests are not being challenged before they reach the content.

Conversely, allowing a crawler does not guarantee that the page will be indexed, selected as a source, quoted, or ranked. Answer engines make their own decisions about relevance, freshness, authority, duplication, and citation.

Make the content easy to discover and cite

  • Use descriptive titles and headings that answer a clear question.
  • Provide stable canonical URLs and useful internal links.
  • Keep important explanatory text in crawlable HTML.
  • Maintain accurate structured data where it matches visible content.
  • Update time-sensitive information and avoid near-duplicate pages.

These practices do not force citation, but they make a public page easier for search systems and users to understand.

Verify the complete path

  1. Validate robots.txt with the checker.
  2. Open the page without authentication and confirm a successful response.
  3. Check WAF rules and official crawler IP verification guidance.
  4. Review logs for allowed crawler requests and response codes.
  5. Allow time for providers to refresh cached policies.

FAQ

Which crawlers should I allow for AI search visibility?

Use each provider’s current documented search token, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, and keep ordinary search crawlers such as Googlebot and Bingbot accessible when organic search matters.

Does Allow: / guarantee that an AI answer engine will cite my page?

No. It permits a compliant crawler to request URLs. Discovery, indexing, selection, ranking, and citation remain decisions made by the provider.

Can I allow AI search but block model-training crawlers?

Yes, when a provider publishes independent controls. For example, OpenAI documents separate GPTBot and OAI-SearchBot tokens, while Anthropic separates ClaudeBot and Claude-SearchBot.

Why is an allowed crawler still receiving errors?

Robots.txt may allow it while a WAF, login, rate limit, IP block, redirect loop, or server error prevents access. Check the full request path and validate the crawler’s source.

Do user-triggered agents determine search visibility?

Not necessarily. Providers publish separate agents for user-requested retrieval, and OpenAI explicitly states that ChatGPT-User is not used to determine Search appearance. Follow each provider’s current documentation.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Website allowing an AI search discovery route while blocking a separate model-training route How to Allow AI Search but Block AI Training

Block only the documented training or product-use controls, such as GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended. Leave AI search crawlers unblocked unless an existing rule blocks them; add explicit search groups only when necessary, and repeat any private-path restrictions inside those groups. Then test the live file by crawler and path.

 AI search crawler discovering public webpages and connecting selected sources to a grounded answer with citations AI Search Crawler: Definition, Visibility, and Control

An AI search crawler discovers and analyzes public web pages for search results, grounded answers, snippets, citations, or source links. Allowing it can support visibility in that AI search product; blocking it may reduce discovery, but neither choice guarantees inclusion or exclusion from every AI surface.

 AI search crawler discovering public website pages and carrying cited sources into a search answer OAI-SearchBot robots.txt Guide

OAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT search features. Allowing it helps OpenAI discover and cite your public pages. Its robots.txt setting is independent from GPTBot, so a site can allow search crawling while opting out of training crawling.

 Public website pages available to Claude search while a separate AI training route is blocked Claude-SearchBot robots.txt Rules

Claude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.

 Perplexity search crawler discovering public websites and carrying selected sources into a cited answer PerplexityBot robots.txt Rules for Search Visibility

PerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.