robots.txt for AI Search and Discovery Crawlers

If AI-search visibility matters, identify the agents used for search/discovery separately from training crawlers. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Applebot have discovery roles, while similarly named training or user-triggered agents may need different policies.

Independent technical reference Updated Editorial approach Report a correction
Public pages moving through AI search discovery and technical access toward possible source citation

Copy-paste robots.txt example

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

“AI crawler” is too broad a category for a useful robots.txt policy. Search/discovery crawlers can help a public site appear in AI-assisted search experiences, while training crawlers and user-triggered fetchers represent different decisions.

Search and discovery agents to know

ProviderSearch/discovery agentSeparate control to avoid confusing with it
OpenAIOAI-SearchBotGPTBot for training-related crawling; ChatGPT-User for user-triggered fetches.
AnthropicClaude-SearchBotClaudeBot for model-development crawling; Claude-User for user-directed retrieval.
PerplexityPerplexityBotPerplexity-User for user-triggered retrieval.
AppleApplebotApplebot-Extended for training data-use control.

Example discovery-first policy

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot
Allow: /

This snippet expresses one narrow idea: allow automatic search/discovery crawlers. It does not make a decision about every training crawler or every user-triggered fetcher.

Example: permit search discovery, restrict selected training crawlers

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

This example expresses four independent preferences, not a universal “block AI” command. OpenAI documents the independent roles of GPTBot and OAI-SearchBot; a specific user-agent group may also stop using restrictions from the wildcard group. If your site blocks /account/ under User-agent: *, duplicate the relevant restriction in every specific group where necessary.

Verification: use the draft Checker with / and /account/profile for each of the four bots. Decide whether the account path should be disallowed before publishing. After publication, check the real robots.txt response and server access logs. Allowing a search crawler is not an indexing or citation guarantee.

Decide training separately

For example, OpenAI documents GPTBot and OAI-SearchBot as independent settings, so you can allow ChatGPT Search crawling while disallowing GPTBot. Apple similarly separates Applebot crawling from the Applebot-Extended data-use signal.

Avoid blanket lists

A copied “block all AI” list can unintentionally remove a site from discovery channels the business actually wants. Start from business intent, map that intent to individual agents, and then publish only the groups required.

Use the robots.txt Generator to build the file and the Checker to test specific paths. For OpenAI details, see OAI-SearchBot; for provider comparisons, see Anthropic agents and Perplexity agents.

Pair crawler policy with technical accessibility

An Allow: / line cannot help if a CDN, WAF or rate-limit rule rejects the crawler before it can fetch the page. When discovery matters, verify both layers: the robots.txt decision and the actual HTTP response to legitimate crawler traffic. Providers that publish verification data should be checked against their current official documentation rather than a permanent hard-coded list copied once.

FAQ

Which crawlers should I allow for AI search visibility?

Use each provider’s current documented search token, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, and keep ordinary search crawlers such as Googlebot and Bingbot accessible when organic search matters.

Does Allow: / guarantee that an AI answer engine will cite my page?

No. It permits a compliant crawler to request URLs. Discovery, indexing, selection, ranking, and citation remain decisions made by the provider.

Can I allow AI search but block model-training crawlers?

Yes, when a provider publishes independent controls. For example, OpenAI documents separate GPTBot and OAI-SearchBot tokens, while Anthropic separates ClaudeBot and Claude-SearchBot.

Why is an allowed crawler still receiving errors?

Robots.txt may allow it while a WAF, login, rate limit, IP block, redirect loop, or server error prevents access. Check the full request path and validate the crawler’s source.

Do user-triggered agents determine search visibility?

Not necessarily. Providers publish separate agents for user-requested retrieval, and OpenAI explicitly states that ChatGPT-User is not used to determine Search appearance. Follow each provider’s current documentation.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Website allowing an AI search discovery route while blocking a separate model-training route How to Allow AI Search but Block AI Training

Block only the documented training or product-use controls, such as GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended. Leave AI search crawlers unblocked unless an existing rule blocks them; add explicit search groups only when necessary, and repeat any private-path restrictions inside those groups. Then test the live file by crawler and path.

 AI search crawler discovering public webpages and connecting selected sources to a grounded answer with citations AI Search Crawler: Definition, Visibility, and Control

An AI search crawler discovers and analyzes public web pages for search results, grounded answers, snippets, citations, or source links. Allowing it can support visibility in that AI search product; blocking it may reduce discovery, but neither choice guarantees inclusion or exclusion from every AI surface.

 AI search crawler discovering public website pages and carrying cited sources into a search answer OAI-SearchBot robots.txt Guide

OAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT search features. Allowing it helps OpenAI discover and cite your public pages. Its robots.txt setting is independent from GPTBot, so a site can allow search crawling while opting out of training crawling.

 Public website pages available to Claude search while a separate AI training route is blocked Claude-SearchBot robots.txt Rules

Claude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.

 Perplexity search crawler discovering public websites and carrying selected sources into a cited answer PerplexityBot robots.txt Rules for Search Visibility

PerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.