GPTBot vs OAI-SearchBot: Training vs Search

GPTBot and OAI-SearchBot are separate OpenAI crawler controls. GPTBot crawls content that may be used to improve and train OpenAI’s generative AI foundation models; OAI-SearchBot discovers pages for ChatGPT search results. You can block one and allow the other.

Independent technical reference Updated Editorial approach Report a correction
Separate GPTBot training and OAI-SearchBot search routes with independent website access controls

Copy-paste robots.txt example

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

The short difference

GPTBot and OAI-SearchBot are not aliases for the same crawler policy. OpenAI publishes them as independent robots.txt controls for different uses. GPTBot relates to content that may contribute to foundation-model training, while OAI-SearchBot supports website discovery for ChatGPT search.

This distinction matters because a full OpenAI block can remove both training access and potential AI-search visibility. A publisher that wants citations and referral opportunities from ChatGPT search but does not want future GPTBot crawling for training should write two separate groups.

GPTBot

  • Purpose: crawl content that may be used to make OpenAI’s generative AI foundation models more useful and safe.
  • Control token: GPTBot.
  • Full-site opt-out: Disallow: / in a GPTBot group.
  • Search effect: blocking GPTBot does not by itself opt the site out of ChatGPT search.

Robots.txt governs future compliant crawling. It is not a mechanism for deleting material already collected, removing an answer, enforcing a license, or blocking unidentified scrapers.

OAI-SearchBot

  • Purpose: surface websites in ChatGPT search features.
  • Control token: OAI-SearchBot.
  • Opt-out effect: OpenAI states that opted-out sites will not be shown in ChatGPT search answers, although navigational links may still appear.
  • Update timing: OpenAI notes that robots.txt changes may take about 24 hours to affect its search systems.

Allowing the bot enables discovery but does not promise indexing, citation, ranking, or traffic for a particular query. The page must still be accessible, useful, technically sound, and relevant to the user’s question.

Side-by-side robots.txt policy

GoalGPTBotOAI-SearchBot
Allow both purposesAllow: /Allow: /
Block training, allow searchDisallow: /Allow: /
Allow training, block searchAllow: /Disallow: /
Block bothDisallow: /Disallow: /

Do not confuse ChatGPT-User with either crawler

ChatGPT-User is used for certain actions initiated by a ChatGPT or Custom GPT user. OpenAI says it is not an automatic web crawler and is not used to determine search inclusion. Because the request is user initiated, robots.txt rules may not apply to it in the same way. Manage automatic ChatGPT search through OAI-SearchBot, not ChatGPT-User.

Recommended configuration and verification

For the common “search yes, training no” policy, publish the example below. Then test the live file with the robots.txt checker, confirm that the file is served from the correct host, and avoid blocking page resources needed for rendering.

OpenAI also publishes IP ranges for its crawlers. When analyzing suspicious requests, verify the current published ranges instead of trusting the User-Agent string alone, which can be spoofed.

For the broader implementation process, use How to Allow AI Search but Block AI Training.

FAQ

Are GPTBot and OAI-SearchBot the same crawler?

No. OpenAI publishes them as independent controls for different purposes: GPTBot for potential foundation-model training use and OAI-SearchBot for ChatGPT search discovery.

Does blocking GPTBot remove my site from ChatGPT search?

Not by itself. Search inclusion is managed with OAI-SearchBot. You can block GPTBot while allowing OAI-SearchBot.

What happens if I block OAI-SearchBot?

OpenAI states that opted-out sites will not be shown in ChatGPT search answers, although they may still appear as navigational links.

How quickly does an OAI-SearchBot rule take effect?

OpenAI notes that it can take approximately 24 hours after a robots.txt update for its search systems to adjust.

Is ChatGPT-User controlled by the OAI-SearchBot rule?

No. ChatGPT-User is a separate user-request agent and is not used to determine search inclusion. OpenAI also notes that robots.txt may not apply to it in the same way.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Website blocking an AI training crawler while keeping a separate search discovery route open GPTBot robots.txt Guide

GPTBot is OpenAI’s automated crawler for content that may be used to improve and train generative AI foundation models. To opt a site out of future GPTBot crawling, add a specific GPTBot group to robots.txt. This setting is separate from OAI-SearchBot and ChatGPT-User.

 AI search crawler discovering public website pages and carrying cited sources into a search answer OAI-SearchBot robots.txt Guide

OAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT search features. Allowing it helps OpenAI discover and cite your public pages. Its robots.txt setting is independent from GPTBot, so a site can allow search crawling while opting out of training crawling.

 Website allowing an AI search discovery route while blocking a separate model-training route How to Allow AI Search but Block AI Training

Block only the documented training or product-use controls, such as GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended. Leave AI search crawlers unblocked unless an existing rule blocks them; add explicit search groups only when necessary, and repeat any private-path restrictions inside those groups. Then test the live file by crawler and path.

 Website selectively blocking AI training crawler routes while keeping search discovery routes open How to Block AI Crawlers with robots.txt

Use the exact published User-agent name for each crawler you intend to restrict, add Disallow: / to that group, preserve any search or user-requested access you still want, then publish and test the live file. Do not use User-agent: * unless you intend to restrict ordinary search crawlers too.

 Website blocking separate OpenAI training, search, and user-request access routes at one policy boundary Block OpenAI Crawlers robots.txt Template

For the broadest OpenAI robots.txt preference, disallow GPTBot, OAI-SearchBot, and ChatGPT-User separately. Expect loss of ChatGPT search visibility and use server controls for user-requested access.