ChatGPT-User, robots.txt, and User-Requested Fetches

ChatGPT-User identifies certain web requests initiated by a person using ChatGPT, a Custom GPT, or an external action. It is not OpenAI’s automatic search or training crawler. OpenAI notes that robots.txt rules may not apply to these user-triggered requests, so private content requires real access controls.

User-initiated AI web request reaching a website protected by server-side access control

Copy-paste robots.txt example

User-agent: ChatGPT-User
Disallow: /

ChatGPT-User is not a conventional crawler

ChatGPT-User is an OpenAI user-agent used for certain actions initiated by a ChatGPT user. A person may ask ChatGPT or a Custom GPT to open a page, retrieve information, or interact with an external application. The resulting request can identify itself as ChatGPT-User.

This behavior differs from a bot that continuously explores the public web. OpenAI explicitly says ChatGPT-User is not used for automatic web crawling. It is also not the control that determines whether pages are eligible for ChatGPT search.

Why robots.txt may not be decisive

Robots.txt was designed primarily for automated crawlers. Because ChatGPT-User requests are triggered by a person, OpenAI states that robots.txt rules may not apply. You can publish a ChatGPT-User group to express a preference, but it should not be treated as a guaranteed access-control barrier.

The sample rule on this page shows the explicit token a site owner might use. Its practical effect can differ from an automatic crawler block, so test the behavior that matters to your site and avoid claims that the rule makes private data inaccessible.

Use the correct OpenAI control

  • OAI-SearchBot: manages automatic crawling for ChatGPT search visibility.
  • GPTBot: manages crawling of content that may be used for foundation-model training.
  • ChatGPT-User: identifies certain user-requested visits and actions.

Blocking ChatGPT-User does not substitute for blocking GPTBot, and it does not opt a page out of ChatGPT search. Those decisions belong in their own user-agent groups.

How to protect content that must stay private

Use authentication, authorization, expiring links, server-side permission checks, and appropriate API security for non-public resources. Do not expose secrets in HTML or rely on a disallow rule to hide a URL. The robots.txt file is public, and a user-directed request can involve behavior outside the assumptions of ordinary crawl management.

For paid content, account dashboards, internal tools, and personal data, the server should deny unauthorized requests regardless of the reported user-agent. User-agent strings can be spoofed, so they are useful for classification and logging, not identity by themselves.

Practical policy and verification

  1. Decide whether user-requested AI access is useful for the page.
  2. Publish a ChatGPT-User rule only as one part of that policy.
  3. Set OAI-SearchBot and GPTBot independently.
  4. Check server logs for the full user-agent and compare source addresses with OpenAI’s current published ranges when verification is necessary.
  5. Test sensitive endpoints as an unauthenticated visitor; they must remain protected even when the user-agent is changed.

The current definitions and network ranges are maintained in the official OpenAI crawler documentation.

FAQ

Is ChatGPT-User used to crawl the web automatically?

No. OpenAI describes ChatGPT-User as a user-agent for certain actions initiated by ChatGPT users, Custom GPTs, or external actions rather than an automatic web crawler.

Will robots.txt always block ChatGPT-User?

No guarantee is stated. OpenAI notes that robots.txt rules may not apply because the requests are initiated by a user. Use server-side access control for anything private.

Does blocking ChatGPT-User remove a site from ChatGPT search?

No. OAI-SearchBot is the robots.txt control for ChatGPT search crawling and search opt-outs.

Does ChatGPT-User control AI training permission?

No. GPTBot is the separate OpenAI crawler used for content that may contribute to foundation-model training.

Can I trust the ChatGPT-User string as proof that a request is from OpenAI?

Not by itself. User-agent strings can be spoofed. Compare the source with OpenAI’s current published IP ranges when authenticity matters.

Related tools

Generate this ruleCheck my robots.txt

Related pages

AI search crawler discovering public website pages and carrying cited sources into a search answer OAI-SearchBot robots.txt Guide

OAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT search features. Allowing it helps OpenAI discover and cite your public pages. Its robots.txt setting is independent from GPTBot, so a site can allow search crawling while opting out of training crawling.

Website blocking an AI training crawler while keeping a separate search discovery route open GPTBot robots.txt Guide

GPTBot is OpenAI’s automated crawler for content that may be used to improve and train generative AI foundation models. To opt a site out of future GPTBot crawling, add a specific GPTBot group to robots.txt. This setting is separate from OAI-SearchBot and ChatGPT-User.

Separate GPTBot training and OAI-SearchBot search routes with independent website access controls GPTBot vs OAI-SearchBot: Training vs Search

GPTBot and OAI-SearchBot are separate OpenAI crawler controls. GPTBot crawls content that may be used to improve and train OpenAI’s generative AI foundation models; OAI-SearchBot discovers pages for ChatGPT search results. You can block one and allow the other.

Robots.txt checker inspecting a website and reporting allowed and blocked crawler access Robots.txt Checker for AI Crawlers

Enter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.

Crawler identities matching specific robots.txt rule groups while unmatched agents follow a separate fallback group User-Agent in robots.txt: Definition and Matching

In robots.txt, User-agent identifies the crawler product token whose rules follow. Matching is case-insensitive. A crawler uses its specifically matching group when available, otherwise it falls back to User-agent: *; repeated groups for the same token are combined.