OAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT search features. Allowing it helps OpenAI discover and cite your public pages. Its robots.txt setting is independent from GPTBot, so a site can allow search crawling while opting out of training crawling.
User-agent: OAI-SearchBot Allow: /
OAI-SearchBot is OpenAI’s search-focused crawler. Its purpose is to discover public pages that may be surfaced, summarized, cited, and linked in ChatGPT search experiences. For publishers that want qualified referral traffic from ChatGPT, this is the OpenAI user-agent that should remain crawlable.
OpenAI treats OAI-SearchBot as a separate control from GPTBot. Allowing OAI-SearchBot does not by itself grant GPTBot permission to crawl the same content for foundation-model training. This separation makes it possible to support search discovery while maintaining a different training policy.
The example on this page explicitly allows OAI-SearchBot. An explicit group is useful when the rest of the file contains broad restrictions or when you want the policy to be easy to audit. After publishing, verify the live result rather than assuming the intended group wins against every other rule in a complex robots.txt file.
OpenAI also publishes IP ranges for OAI-SearchBot. When logs or firewall rules matter, validate both the user-agent and the current published network ranges instead of trusting the user-agent string alone.
OpenAI’s crawler documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although a page can still appear as a navigational link when OpenAI learns its URL from another source. Blocking the crawler therefore affects discovery and answer inclusion, but it is not identical to a universal removal from every possible link surface.
If you do not want a page indexed or surfaced, use the appropriate noindex directive as well. The crawler must be allowed to fetch a page before it can read a page-level robots meta tag, so blocking the URL in robots.txt and expecting the bot to discover noindex can be contradictory.
A typical policy for a publisher that wants ChatGPT referrals but not training use is:
The GPTBot vs OAI-SearchBot guide explains the difference in more detail, and the allow search, block training template provides a starting policy.
utm_source=chatgpt.com to referral URLs from ChatGPT search results.See the official OpenAI crawler documentation for current user-agent strings and verification ranges.
Not by itself. OAI-SearchBot and GPTBot are independent robots.txt controls. Manage training crawling with the GPTBot group.
Not necessarily. OpenAI says a blocked page may still appear as a navigational link when its URL is learned from another source, but it will not be included normally in ChatGPT search answers.
Use noindex when you do not want a page indexed or surfaced. Remember that the crawler must be allowed to fetch the page before it can read a page-level noindex directive.
OpenAI says search systems may take about 24 hours to adjust after a robots.txt update. Server and cache behavior can also affect when the live file becomes visible.
OpenAI currently adds utm_source=chatgpt.com to referral URLs. Use that parameter in your analytics reports to identify visits from ChatGPT search results.
GPTBot robots.txt GuideGPTBot is OpenAI’s automated crawler for content that may be used to improve and train generative AI foundation models. To opt a site out of future GPTBot crawling, add a specific GPTBot group to robots.txt. This setting is separate from OAI-SearchBot and ChatGPT-User.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
GPTBot vs OAI-SearchBot: Training vs SearchGPTBot and OAI-SearchBot are separate OpenAI crawler controls. GPTBot crawls content that may be used to improve and train OpenAI’s generative AI foundation models; OAI-SearchBot discovers pages for ChatGPT search results. You can block one and allow the other.
robots.txt for AI Search Visibility and CitationsTo support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.