An AI crawler is a web agent used by an AI provider to collect, discover, index, or retrieve online content. The label is broad: training crawlers, AI search crawlers, and user-requested fetchers can have different purposes and separate robots.txt tokens.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /
An AI crawler is an automated or semi-automated web agent operated in connection with an AI product. It may collect public content for model development, build a search index, refresh source data, or retrieve a page after a user asks an assistant to access it. There is no single industry-wide “AI crawler” user-agent.
The practical unit of control is the documented token published by the operator: examples include GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, and others. These tokens are not interchangeable, and some are policy controls rather than independent HTTP crawlers.
These agents collect public web content that may contribute to training, improvement, evaluation, or safety work. Site owners often choose to block them while keeping ordinary search engines and AI search discovery open.
These agents discover and analyze pages for search-style results, grounded answers, citations, and source links. Blocking them may reduce visibility in the provider’s AI search experience, but allowing them does not guarantee inclusion or citation.
These agents access a page because a person asked an assistant or tool to retrieve it. Their behavior and robots.txt support can differ from automated crawlers. Private content still requires real authentication and authorization.
A broad rule such as User-agent: * Disallow: / can block ordinary search crawling along with AI agents. A more precise policy identifies the specific purpose you want to restrict. For example, you can disallow a training crawler and allow the provider’s separate search crawler.
Operator documentation can change, so review the current crawler list before publishing a policy. The AI crawler directory organizes documented agents by provider and purpose.
Compliant crawlers fetch the robots.txt file for the exact scheme, host, and port they are visiting. They select the most specific matching user-agent group and apply the relevant Allow and Disallow rules. Each subdomain needs its own file.
Robots.txt is not an access-control system. It cannot protect customer records, paid content, unpublished documents, or internal APIs from a crawler that ignores the protocol. Use login controls, authorization, signed URLs, network rules, and rate limiting where access must be enforced.
No. AI providers publish separate tokens for different products and purposes, and those tokens must be controlled individually.
No. Some agents support model development, others power AI search discovery, and others retrieve a page after a user request.
Often yes, when the provider publishes separate user-agent tokens. Configure specific groups rather than using a broad wildcard block.
No. It is a voluntary crawl policy. Private content requires authentication, authorization, network controls, or other server-side enforcement.
Compare the full User-Agent and network source with the operator’s current documentation or published IP verification method. User-Agent text alone can be spoofed.
AI Crawler ListThis directory groups crawler tokens by purpose so you can decide what to allow or block without treating every automated request as the same. Open a crawler page for its documented role, exact robots.txt token, policy trade-offs, and verification guidance.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
AI Training Crawler: Definition and robots.txt ControlAn AI training crawler collects public web content that may contribute to model development, improvement, evaluation, or safety work. A crawler-specific Disallow rule can signal that future collection is not permitted, but it does not erase previously acquired data or secure private content.
AI Search Crawler: Definition, Visibility, and ControlAn AI search crawler discovers and analyzes public web pages for search results, grounded answers, snippets, citations, or source links. Allowing it can support visibility in that AI search product; blocking it may reduce discovery, but neither choice guarantees inclusion or exclusion from every AI surface.
