An AI training crawler collects public web content that may contribute to model development, improvement, evaluation, or safety work. A crawler-specific Disallow rule can signal that future collection is not permitted, but it does not erase previously acquired data or secure private content.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: /
An AI training crawler is a web agent that gathers public online material for possible use in developing or improving generative AI systems. Depending on the operator, collected content may support training, fine-tuning, evaluation, safety research, data filtering, or related model-development work.
The term describes a purpose, not one universal protocol token. GPTBot and ClaudeBot are documented examples of model-development crawlers. Other controls, such as Google-Extended or Applebot-Extended, may act as product-use tokens rather than separate HTTP crawlers. Always follow the operator’s current documentation.
An AI search crawler discovers pages for search results, grounded answers, snippets, citations, or source links. A training crawler collects content for model-development purposes. Some providers publish separate agents so a site can allow search discovery while opting out of training collection.
User-requested fetchers are a third category. They may access a page because a person asked an assistant to retrieve it. Their robots.txt behavior can differ from automated training and search agents.
Create a specific group for each documented training token and disallow the required scope. Blocking / requests that the crawler avoid the entire host. A narrower path rule can keep public marketing pages available while excluding archives, user profiles, or other sections.
Do not rely on a generic guessed token such as “AI-bot.” Crawlers match documented user-agent groups, and an invented name controls nothing. Use the allow search, block AI training template as a starting point.
Robots.txt is prospective crawl guidance. It does not automatically delete content already collected, remove copies held by third parties, revoke licensed datasets, or control information obtained through another source. Provider-specific opt-out forms, contractual rights, or legal requests may be separate processes.
It is also not security. If information must remain private, protect it with authentication, authorization, network controls, or access-restricted storage. Publicly accessible URLs should be treated as public even when a polite crawler is disallowed.
Blocking a dedicated training crawler should not block ordinary search engines unless your file also targets Googlebot, Bingbot, or a broad wildcard group. Review every group carefully. A misplaced User-agent: * rule can harm search visibility far beyond the intended AI-training policy.
When a provider separates training and search agents, keep the search agent open if AI search visibility and citations are valuable to the site. See How to Allow AI Search but Block AI Training.
It gathers public web content that may contribute to model training, improvement, evaluation, safety, or related development work.
No. Training crawlers support model development, while AI search crawlers discover sources for search results, grounded answers, or citations.
No. Robots.txt controls future compliant crawling. It does not automatically erase earlier copies, licensed datasets, or data obtained from other sources.
Yes when the provider publishes separate tokens. Disallow the training agent and explicitly keep the search agent accessible.
Not by itself. Google Search uses Googlebot. Search visibility is at risk only if Googlebot or an overly broad wildcard group is also blocked.
GPTBot robots.txt GuideGPTBot is OpenAI’s automated crawler for content that may be used to improve and train generative AI foundation models. To opt a site out of future GPTBot crawling, add a specific GPTBot group to robots.txt. This setting is separate from OAI-SearchBot and ChatGPT-User.
ClaudeBot robots.txt GuideClaudeBot is Anthropic’s crawler for public web content that may contribute to model development and training. A ClaudeBot-specific Disallow rule signals that future material should be excluded from Anthropic training crawls. Claude-SearchBot and Claude-User are separate agents with different purposes.
Google-Extended robots.txt Control for GeminiGoogle-Extended is a standalone robots.txt control token for Gemini model training and grounding. Blocking it does not block Googlebot, remove pages from Google Search, or act as a Google Search ranking signal.
robots.txt for AI Training: Build a Selective Opt-OutTo restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.
Allow Search, Block AI Training robots.txt TemplateUse separate user-agent groups: block documented training or dataset agents such as GPTBot and ClaudeBot, while allowing ordinary search and dedicated AI-search crawlers.
