To support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: /
AI answer engines do not all use one universal crawler. OpenAI documents OAI-SearchBot for ChatGPT search, Anthropic documents Claude-SearchBot for search experiences, and Perplexity documents PerplexityBot for surfacing and linking websites in its results. These agents are separate from training-oriented and user-triggered agents.
If you want your public pages to be eligible for discovery, address the exact search token rather than allowing every crawler from the provider.
A visibility-oriented file can make search access clear:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /Explicit Allow: / groups are useful when a broader wildcard group blocks other agents. Review the whole file because a crawler selects the most specific matching user-agent group and then evaluates the applicable path rules. A rule that looks correct in isolation may still be defeated by a typo, duplicate group, or more specific path restriction.
You can allow AI search discovery while restricting model-development agents:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /This does not create a universal legal or technical policy, but it records a clear provider-specific preference. Use the selective AI policy guide for a larger multi-provider example.
An allowed crawler can still fail to reach a page because of a login, WAF challenge, blocked IP range, repeated server errors, redirect loop, or inaccessible rendering resources. Confirm that the page returns a stable successful response and that legitimate crawler requests are not being challenged before they reach the content.
Conversely, allowing a crawler does not guarantee that the page will be indexed, selected as a source, quoted, or ranked. Answer engines make their own decisions about relevance, freshness, authority, duplication, and citation.
These practices do not force citation, but they make a public page easier for search systems and users to understand.
Use each provider’s current documented search token, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, and keep ordinary search crawlers such as Googlebot and Bingbot accessible when organic search matters.
No. It permits a compliant crawler to request URLs. Discovery, indexing, selection, ranking, and citation remain decisions made by the provider.
Yes, when a provider publishes independent controls. For example, OpenAI documents separate GPTBot and OAI-SearchBot tokens, while Anthropic separates ClaudeBot and Claude-SearchBot.
Robots.txt may allow it while a WAF, login, rate limit, IP block, redirect loop, or server error prevents access. Check the full request path and validate the crawler’s source.
Not necessarily. Providers publish separate agents for user-requested retrieval, and OpenAI explicitly states that ChatGPT-User is not used to determine Search appearance. Follow each provider’s current documentation.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
AI Search Crawler: Definition, Visibility, and ControlAn AI search crawler discovers and analyzes public web pages for search results, grounded answers, snippets, citations, or source links. Allowing it can support visibility in that AI search product; blocking it may reduce discovery, but neither choice guarantees inclusion or exclusion from every AI surface.
OAI-SearchBot robots.txt GuideOAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT search features. Allowing it helps OpenAI discover and cite your public pages. Its robots.txt setting is independent from GPTBot, so a site can allow search crawling while opting out of training crawling.
Claude-SearchBot robots.txt RulesClaude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.
PerplexityBot robots.txt Rules for Search VisibilityPerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.