GPTBot and OAI-SearchBot are separate OpenAI crawler controls. GPTBot crawls content that may be used to improve and train OpenAI’s generative AI foundation models; OAI-SearchBot discovers pages for ChatGPT search results. You can block one and allow the other.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /
GPTBot and OAI-SearchBot are not aliases for the same crawler policy. OpenAI publishes them as independent robots.txt controls for different uses. GPTBot relates to content that may contribute to foundation-model training, while OAI-SearchBot supports website discovery for ChatGPT search.
This distinction matters because a full OpenAI block can remove both training access and potential AI-search visibility. A publisher that wants citations and referral opportunities from ChatGPT search but does not want future GPTBot crawling for training should write two separate groups.
GPTBot.Disallow: / in a GPTBot group.Robots.txt governs future compliant crawling. It is not a mechanism for deleting material already collected, removing an answer, enforcing a license, or blocking unidentified scrapers.
OAI-SearchBot.Allowing the bot enables discovery but does not promise indexing, citation, ranking, or traffic for a particular query. The page must still be accessible, useful, technically sound, and relevant to the user’s question.
| Goal | GPTBot | OAI-SearchBot |
|---|---|---|
| Allow both purposes | Allow: / | Allow: / |
| Block training, allow search | Disallow: / | Allow: / |
| Allow training, block search | Allow: / | Disallow: / |
| Block both | Disallow: / | Disallow: / |
ChatGPT-User is used for certain actions initiated by a ChatGPT or Custom GPT user. OpenAI says it is not an automatic web crawler and is not used to determine search inclusion. Because the request is user initiated, robots.txt rules may not apply to it in the same way. Manage automatic ChatGPT search through OAI-SearchBot, not ChatGPT-User.
For the common “search yes, training no” policy, publish the example below. Then test the live file with the robots.txt checker, confirm that the file is served from the correct host, and avoid blocking page resources needed for rendering.
OpenAI also publishes IP ranges for its crawlers. When analyzing suspicious requests, verify the current published ranges instead of trusting the User-Agent string alone, which can be spoofed.
For the broader implementation process, use How to Allow AI Search but Block AI Training.
No. OpenAI publishes them as independent controls for different purposes: GPTBot for potential foundation-model training use and OAI-SearchBot for ChatGPT search discovery.
Not by itself. Search inclusion is managed with OAI-SearchBot. You can block GPTBot while allowing OAI-SearchBot.
OpenAI states that opted-out sites will not be shown in ChatGPT search answers, although they may still appear as navigational links.
OpenAI notes that it can take approximately 24 hours after a robots.txt update for its search systems to adjust.
No. ChatGPT-User is a separate user-request agent and is not used to determine search inclusion. OpenAI also notes that robots.txt may not apply to it in the same way.
GPTBot robots.txt GuideGPTBot is OpenAI’s automated crawler for content that may be used to improve and train generative AI foundation models. To opt a site out of future GPTBot crawling, add a specific GPTBot group to robots.txt. This setting is separate from OAI-SearchBot and ChatGPT-User.
OAI-SearchBot robots.txt GuideOAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT search features. Allowing it helps OpenAI discover and cite your public pages. Its robots.txt setting is independent from GPTBot, so a site can allow search crawling while opting out of training crawling.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
Block OpenAI Crawlers robots.txt TemplateFor the broadest OpenAI robots.txt preference, disallow GPTBot, OAI-SearchBot, and ChatGPT-User separately. Expect loss of ChatGPT search visibility and use server controls for user-requested access.