GPTBot is OpenAI’s automated crawler for content that may be used to improve and train generative AI foundation models. To opt a site out of future GPTBot crawling, add a specific GPTBot group to robots.txt. This setting is separate from OAI-SearchBot and ChatGPT-User.
User-agent: GPTBot Disallow: /
GPTBot is OpenAI’s automated web crawler for content that may contribute to the development, usefulness, and safety of generative AI foundation models. OpenAI documents GPTBot separately from its search crawler and user-triggered agent, which means site owners can set a training preference without automatically removing the site from ChatGPT search.
The important distinction is purpose: GPTBot is the training-oriented crawler. OAI-SearchBot is used for ChatGPT search visibility, while ChatGPT-User may fetch a page after a person asks ChatGPT or a Custom GPT to access it.
To disallow GPTBot across the entire host, publish the rule shown in the example on this page. The file must be available at the root of that exact host, such as https://example.com/robots.txt. If the same business uses separate subdomains, each subdomain can have its own robots.txt policy.
You can also disallow selected paths instead of the whole site. For example, a publisher might allow public articles while disallowing an archive or member area. However, robots.txt is a public crawling instruction, not a privacy control. Sensitive pages still need authentication and appropriate server-side protection.
OpenAI states that GPTBot and OAI-SearchBot are independent controls. A common publisher policy is therefore to block GPTBot while allowing OAI-SearchBot. That combination expresses two different choices: do not use future crawled content for foundation-model training, but continue allowing the site to be discovered and cited in ChatGPT search features.
Do not use a broad User-agent: * block unless you actually intend to affect regular search engines and every other compliant crawler. A crawler-specific group is safer when your goal concerns GPTBot alone.
Disallowing GPTBot signals that material from the blocked host or path should not be used in future foundation-model training crawls. It does not remove information already published elsewhere, delete existing model knowledge, prevent people from copying a page, or secure a private resource. For those goals, use access control, contractual measures, and content-removal procedures in addition to robots.txt.
OpenAI’s current crawler definitions and published verification ranges are available in the official OpenAI crawler documentation.
No. GPTBot and OAI-SearchBot are separate controls. You can block GPTBot for training purposes while allowing OAI-SearchBot for ChatGPT search visibility.
No. A robots.txt rule controls future compliant crawling. It is not a retroactive deletion request and does not remove copies held by other websites or services.
Yes. Use path-specific Disallow rules under the GPTBot group. Private or sensitive folders must still be protected with authentication because robots.txt is publicly readable.
Usually not when GPTBot is your only concern. A wildcard block may also affect search engines and other compliant crawlers. Use the specific GPTBot token for a targeted policy.
Check the request against OpenAI’s current published IP ranges and review the user-agent in your server logs. A user-agent string by itself can be spoofed.
OAI-SearchBot robots.txt GuideOAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT search features. Allowing it helps OpenAI discover and cite your public pages. Its robots.txt setting is independent from GPTBot, so a site can allow search crawling while opting out of training crawling.
Block GPTBot robots.txt TemplateAdd a dedicated GPTBot group with Disallow: /. This blocks OpenAI's training-oriented crawler without automatically blocking OAI-SearchBot or ordinary search engines.
Block OpenAI Crawlers robots.txt TemplateFor the broadest OpenAI robots.txt preference, disallow GPTBot, OAI-SearchBot, and ChatGPT-User separately. Expect loss of ChatGPT search visibility and use server controls for user-requested access.
GPTBot vs OAI-SearchBot: Training vs SearchGPTBot and OAI-SearchBot are separate OpenAI crawler controls. GPTBot crawls content that may be used to improve and train OpenAI’s generative AI foundation models; OAI-SearchBot discovers pages for ChatGPT search results. You can block one and allow the other.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
