AI Crawler List and robots.txt Directory

This directory groups crawler tokens by purpose so you can decide what to allow or block without treating every automated request as the same. Open a crawler page for its documented role, exact robots.txt token, policy trade-offs, and verification guidance.

Crawler directory separating training, AI search, user-requested, conventional search, and SEO crawler purposes

Use purpose, not the word “AI,” to choose a policy

Crawler names alone do not tell you what a request is for. Providers increasingly publish separate tokens for model development, search discovery, and requests initiated by a user. Traditional search engines, open datasets, and SEO platforms add further categories. A useful robots.txt policy starts by identifying the purpose you want to allow or restrict.

Categories in this directory

  • AI training and model-development crawlers collect public material that may contribute to future model improvement. Examples in the directory include GPTBot and ClaudeBot.
  • AI search crawlers discover and index pages for search-style answers and citations, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot.
  • User-triggered agents retrieve a page after a person asks an assistant to access it. Provider treatment of robots.txt is not identical, so read the individual page before relying on a rule.
  • Search-engine crawlers such as Googlebot, Bingbot, and Applebot support conventional or platform search discovery.
  • Dataset and SEO crawlers include CCBot, AhrefsBot, and SemrushBot. Their purpose and the benefit of allowing them differ from consumer search.
  • Product-control tokens such as Google-Extended and Applebot-Extended express preferences about specified content uses but are not necessarily independent HTTP crawlers.

How to use a crawler entry

  1. Confirm the exact token and purpose from the provider’s current documentation.
  2. Decide whether the business goal is search visibility, training opt-out, user-directed access, analytics, or server-load control.
  3. Add the narrowest rule that expresses that decision.
  4. Publish the file at the root of the correct host and protocol.
  5. Test the live result with the checker and verify suspicious traffic using provider-published IP or DNS information when available.

The directory is a maintained working reference, not a claim that every crawler on the web is listed. Providers can add tokens or change behavior, and a User-Agent header can be spoofed. Use robots.txt for cooperative crawling policy and real access controls for sensitive resources.

Crawler directory

Bot Company Category Token Recommended action
AhrefsBot Ahrefs seo crawler AhrefsBot Optional: allow for SEO tools, block if server load or data access is a concern.
Claude-SearchBot Anthropic ai search crawler Claude-SearchBot Allow if you want Claude search features to discover your pages.
Claude-User Anthropic user triggered agent Claude-User Allow if user-triggered Claude access should work.
ClaudeBot Anthropic ai training crawler ClaudeBot Block if you want to restrict Anthropic training crawler access.
Applebot Apple search crawler Applebot Usually allow if you want Apple search and assistant integrations to discover your pages.
Applebot-Extended Apple product token Applebot-Extended Block if you want to signal restrictions for Apple AI-related use while treating Applebot separately.
CCBot Common Crawl archive crawler CCBot Block if you do not want your pages included in Common Crawl datasets.
Google-Extended Google product token Google-Extended Block if you want to opt out of certain Google AI product training/use signals while keeping Google Search crawling separate.
Googlebot Google search crawler Googlebot Do not block unless you intentionally want to remove regular Google crawling.
MJ12bot Majestic seo crawler MJ12bot Optional: allow for link data, block if crawler load is not wanted.
Bingbot Microsoft search crawler Bingbot Do not block unless you intentionally want to remove Bing crawling.
ChatGPT-User OpenAI user triggered agent ChatGPT-User Allow if users should be able to ask ChatGPT to fetch your pages.
GPTBot OpenAI ai training crawler GPTBot Block if you want to signal that your public content should not be used for AI training.
OAI-SearchBot OpenAI ai search crawler OAI-SearchBot Allow if you want visibility in OpenAI search products.
Perplexity-User Perplexity user triggered agent Perplexity-User Allow if user-triggered Perplexity access should work.
PerplexityBot Perplexity ai search crawler PerplexityBot Allow if you want discovery in Perplexity search; block if you do not want this crawler.
SemrushBot Semrush seo crawler SemrushBot Optional: allow for SEO tools, block if server load or data access is a concern.

FAQ

Is every bot in this directory an AI training crawler?

No. The directory includes training crawlers, AI search crawlers, user-triggered agents, conventional search crawlers, dataset crawlers, SEO crawlers, and product-control tokens.

Can one robots.txt rule block every crawler from a provider?

Not reliably. Many providers document multiple independent tokens for different purposes. Use the exact user-agent groups that match the access you want to control.

Does allowing an AI search crawler also allow model training?

Not necessarily. Some providers publish independent search and training controls. Review the individual crawler documentation instead of assuming one token governs every use.

How can I tell whether a logged crawler is genuine?

Do not trust the User-Agent string alone. Where available, compare the request with the provider’s published IP ranges or perform the documented forward and reverse DNS checks.

Why are Google-Extended and Applebot-Extended listed if they do not crawl independently?

They are robots.txt control tokens that affect specified downstream uses of content collected by associated systems. Their individual pages explain the distinction from a normal HTTP crawler.