An AI search crawler discovers and analyzes public web pages for search results, grounded answers, snippets, citations, or source links. Allowing it can support visibility in that AI search product; blocking it may reduce discovery, but neither choice guarantees inclusion or exclusion from every AI surface.
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
An AI search crawler explores public web pages to build or refresh a search index, identify relevant sources, and support grounded answers with links or citations. Examples of documented search-focused agents include OAI-SearchBot, Claude-SearchBot, and PerplexityBot.
These agents are connected to AI search experiences, but they are still web crawlers. They fetch URLs, follow links, interpret page content, and revisit sources according to the operator’s scheduling and policies.
A training crawler collects material that may contribute to model development. A search crawler discovers sources for current retrieval, ranking, grounding, or citation. Providers may publish separate user-agent tokens so a publisher can allow search discovery while disallowing model-training collection.
A user-requested fetcher is different again: it may visit one page because a person asked an assistant to open, summarize, or use it. Do not assume that a search crawler rule automatically controls the provider’s user-directed agent.
Allowing an AI search crawler makes your public pages available for that crawler to inspect. This can improve the chance that the provider discovers, refreshes, and cites the content. It does not guarantee ranking, citation, traffic, or inclusion in a generated answer.
Clear titles, descriptive headings, crawlable internal links, stable canonical URLs, useful page content, and accessible source information all remain important. A sitemap can support discovery, but it does not override a robots.txt block.
A crawler-specific Disallow: / asks that search agent not to crawl the host. The provider may have less direct access to the page content, which can reduce freshness, visibility, or citation accuracy in that product. Exact behavior varies by provider.
Blocking an AI search crawler does not automatically remove a URL from Google or Bing, and it does not necessarily prevent a provider from knowing the URL through third-party search data, links, or previous crawls. If a page must not be publicly accessible, use real access control.
When AI search visibility is useful, allow documented search agents and block only the model-development agents you do not want. Avoid broad wildcard rules that also block ordinary search engines. The selective template provides a safe starting structure.
Review each provider separately. See the guides for GPTBot versus OAI-SearchBot, ClaudeBot versus Claude-SearchBot, and PerplexityBot versus Perplexity-User.
It discovers and analyzes public web pages for search results, grounded answers, source links, snippets, or citations in an AI search product.
Not necessarily. Providers often publish separate agents for search discovery and model development. Check the official purpose of each token.
No. It makes crawling possible, but discovery, ranking, citation, and traffic remain subject to the provider’s systems and the quality and relevance of the page.
No, not by itself. Google Search and Bing use their own crawlers. Only rules that target those crawlers or a broad wildcard group affect their crawl access.
Yes when separate tokens are available. Allow the provider’s search crawler and disallow its model-development crawler in distinct robots.txt groups.
OAI-SearchBot robots.txt GuideOAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT search features. Allowing it helps OpenAI discover and cite your public pages. Its robots.txt setting is independent from GPTBot, so a site can allow search crawling while opting out of training crawling.
Claude-SearchBot robots.txt RulesClaude-SearchBot is Anthropic’s automated search crawler. Allowing it can help public pages appear in Claude search results; blocking it can reduce that visibility. Control it separately from ClaudeBot and Claude-User.
PerplexityBot robots.txt Rules for Search VisibilityPerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.
robots.txt for AI Search Visibility and CitationsTo support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
