Crawl-delay asks a crawler to wait between requests, but it is a non-standard robots.txt extension and support varies by crawler. Use it only for a documented user-agent and only when server logs show that crawl rate is a real problem.
User-agent: Bingbot Allow: / Crawl-delay: 5 User-agent: ClaudeBot Disallow: /private/ Crawl-delay: 5
Crawl-delay is a robots.txt extension intended to reduce how frequently a crawler requests pages from a host. A typical value represents a delay in seconds, but the exact interpretation and allowed range depend on the crawler. It is not part of the core Robots Exclusion Protocol standardized by RFC 9309, so there is no universal behavior.
This makes support the first question to answer. Bing documents that Bingbot honors Crawl-delay. Some AI crawler operators also document support for their own agents. Other crawlers ignore it, and Googlebot is not configured through this directive. Always consult the current documentation for the exact user-agent you are targeting.
Place the directive inside the relevant User-agent group. Do not add it under User-agent: * unless you intentionally want every crawler that recognizes the extension to inherit the delay.
User-agent: Bingbot
Crawl-delay: 5A targeted group is safer because search crawlers, SEO audit bots, archive crawlers, and AI crawlers can have different traffic patterns and different support rules.
An aggressive value can slow discovery of new pages, price changes, corrections, deleted URLs, canonical updates, and sitemap changes. On a large site, a crawler may need days or weeks longer to revisit important content. That can be more damaging than a short temporary increase in server load.
Use access rules for URLs that should not be crawled at all. Use Crawl-delay only for a crawler that should retain access but needs to request pages more slowly. These are separate policy decisions.
Before adding a delay, inspect server logs. Confirm the crawler identity, request volume, response times, status codes, and the paths causing load. Many crawl problems are actually caused by infinite URL spaces, faceted navigation, session parameters, slow templates, redirect chains, or error responses.
For Googlebot, use Google’s documented crawl-rate and host-load guidance rather than relying on Crawl-delay. For Bingbot, Bing Webmaster Tools provides crawl controls in addition to robots.txt. If the traffic is malicious or ignores robots.txt, use rate limiting, authentication, firewall rules, or application-level controls instead.
User-agent: *.No. RFC 9309 standardizes core robots.txt behavior, while Crawl-delay is a non-standard extension. Support and interpretation vary by crawler.
Googlebot is not controlled through the Crawl-delay directive. Use Google’s documented crawl-rate and host-load guidance instead.
Yes. Bing documents support for Crawl-delay, but a high value can significantly slow discovery and recrawling.
Usually not. A crawler-specific group is safer because wildcard placement may affect several crawlers that recognize the extension.
No. It is a voluntary instruction for compliant crawlers. Use server-side rate limiting, authentication, firewall controls, or blocking for abusive traffic.
Bingbot robots.txt Rules for Bing SearchBingbot is Microsoft Bing’s primary search crawler. It follows Bingbot-specific robots.txt groups and supports Crawl-delay. Blocking a path limits crawling but does not reliably remove an already known URL from Bing, so use page-level noindex or Bing removal tools when exclusion from search results is the real goal.
AhrefsBot robots.txt Rules and Crawl ControlAhrefsBot follows links to build Ahrefs’ web and backlink index. Ahrefs says it strictly respects Allow, Disallow, and Crawl-delay rules, so you can block the crawler or reduce its request frequency without blocking search engines.
SemrushBot robots.txt Rules for SEO CrawlingSemrushBot is Semrush’s crawler for backlink and webgraph data. You can block it with a SemrushBot group or use Crawl-delay up to Semrush’s documented 10-second limit, but other Semrush tools use separate crawler tokens.
robots.txt Definition, Syntax, and ExamplesRobots.txt is a UTF-8 plain-text file served at the top-level /robots.txt path of a host. It groups crawler product tokens under User-agent lines and uses Allow and Disallow rules to request which URL paths compliant crawlers may access.
Common robots.txt Mistakes and How to Fix ThemThe most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.