Block OpenAI Crawlers robots.txt Template

Strict OpenAI robots.txt preference

OpenAI publishes three relevant user-agent identities with different purposes. The following template expresses the broadest robots.txt preference by disallowing all three while leaving unrelated crawlers open:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: *
Allow: /

Use this only when you are prepared to give up OpenAI training crawls and ChatGPT search discovery. The ChatGPT-User group records a preference, but user-initiated fetches need additional controls because OpenAI states that robots.txt may not apply to them.

GPTBot: model-training collection

GPTBot is used to crawl content that may contribute to OpenAI's generative AI foundation-model training. Blocking this token is the targeted option when training is your only concern. It does not automatically block ChatGPT search.

OAI-SearchBot: ChatGPT search visibility

OAI-SearchBot is used to surface sites in ChatGPT search features. Disallowing it can prevent pages from being shown in ChatGPT search answers, although navigational links may still appear. OpenAI treats this setting independently from GPTBot, so you can block one and allow the other.

ChatGPT-User: user-initiated access

ChatGPT-User can visit a page because a person asked ChatGPT or a Custom GPT to access it. It is not an automatic search crawler and does not determine search inclusion. OpenAI notes that robots.txt rules may not apply to these user actions. If a page must not be retrieved, require authentication or enforce access at the application, server, CDN, or WAF layer.

Choose a narrower policy when possible

Many public sites do not need the full block. Common alternatives are:

• Block training only: disallow GPTBot and allow OAI-SearchBot.
• Block ChatGPT search only: disallow OAI-SearchBot while making a separate decision about GPTBot.
• Protect selected areas: use path-specific rules and real authorization for private content.

The GPTBot vs OAI-SearchBot comparison explains the trade-off, and the selective template provides a less restrictive configuration.

Merge the rules with the existing file

Do not upload the sample as a blind replacement if the site already has path restrictions, sitemap declarations, or crawler-specific allowances. Merge the three OpenAI groups into the current file, remove duplicate groups, and confirm that the final response still represents the intended policy for Googlebot, Bingbot, images, and other assets.

Publish and verify the file

1. Merge the groups into the existing robots.txt file at the host root.
2. Preserve sitemap lines and unrelated crawler rules that are still needed.
3. Test the homepage and representative paths for each OpenAI token with the checker.
4. Allow time for OpenAI systems to refresh the policy.
5. Use official IP-range data when verifying whether logged requests are genuine.

This template does not include OAI-AdsBot, which OpenAI documents for validating landing pages submitted as ChatGPT ads rather than general automatic crawling of a public site.