AhrefsBot follows links to build Ahrefs’ web and backlink index. Ahrefs says it strictly respects Allow, Disallow, and Crawl-delay rules, so you can block the crawler or reduce its request frequency without blocking search engines.
User-agent: AhrefsBot Disallow: /
AhrefsBot is the web crawler Ahrefs uses to discover pages and links for its web index and backlink-analysis products. It follows links found on the internet, requests pages with HTTP GET, and periodically revisits known URLs to check their current status. The crawler does not represent Google Search and has no direct control over Google or Bing rankings.
Ahrefs says AhrefsBot strictly respects both Allow and Disallow directives. To stop it from crawling the entire host, publish the two-line group shown below. To exclude only expensive or low-value sections, replace / with specific directory paths.
Changes may not take effect immediately. AhrefsBot reads updated robots.txt instructions before a later scheduled crawl, so requests can continue for a period after you publish a new rule. Check the timestamp of the live file before treating continued requests as noncompliance.
If bandwidth or server load is the concern, you can use the non-standard Crawl-delay directive instead of a full block. Ahrefs documents the value as the minimum delay, in seconds, between consecutive requests from AhrefsBot. Start conservatively and monitor real server metrics rather than adding a very large delay without evidence.
Ahrefs operates more than one crawler. AhrefsBot builds the broader Ahrefs web index, while AhrefsSiteAudit crawls sites for the Site Audit product. A rule for AhrefsBot does not automatically control AhrefsSiteAudit. Verified site owners can also configure AhrefsSiteAudit to ignore robots.txt for their own project, which is a product setting and not normal public crawling behavior.
Blocking AhrefsBot can reduce or eventually remove your site’s presence in Ahrefs’ link index and reports. It does not directly prevent Googlebot from crawling the site. Keep search-engine rules in separate groups, and avoid a broad User-agent: * block unless you intend to restrict all compliant crawlers.
Ahrefs publishes crawler user agents and IP information for verification. If traffic continues, first confirm that robots.txt returns successfully from the correct host, contains valid syntax, and targets the exact crawler token. Then check the file with the checker. For general syntax risks, review common robots.txt mistakes and the Crawl-delay glossary.
AhrefsBot follows web links and revisits pages to build and maintain the Ahrefs web and backlink index.
Ahrefs says AhrefsBot strictly respects Allow, Disallow, and Crawl-delay directives.
Yes. Add Crawl-delay under the AhrefsBot group and specify the minimum interval in seconds between requests.
No. AhrefsBot builds the broader Ahrefs index, while AhrefsSiteAudit is a separate crawler used for site-audit projects.
Not directly. Googlebot is a separate crawler. Blocking AhrefsBot mainly affects Ahrefs data and reports unless broader robots.txt rules also restrict search engines.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
Common robots.txt Mistakes and How to Fix ThemThe most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.
How to Check Whether AI Crawlers Are BlockedFetch the live robots.txt file, select the crawler’s exact product token, and test the specific URL against the matching group and longest Allow or Disallow rule. Then check HTTP responses, caching, firewall rules, and logs, because a crawler may be allowed by robots.txt yet blocked by the server—or disallowed by robots.txt while an unidentified scraper still reaches the page.
robots.txt HTTP Status Codes: 200, 3xx, 4xx, and 5xxA successful 2xx response lets crawlers parse robots.txt. Redirects may be followed, ordinary 4xx responses often mean no usable robots.txt restrictions, while 5xx and network failures are treated as temporary retrieval problems. Exact behavior and cache timing vary by crawler, so return a stable 200 response for a valid file and never use 401 or 403 as a substitute for crawl rules.
Crawl-delay in robots.txt: Meaning and Crawler SupportCrawl-delay asks a crawler to wait between requests, but it is a non-standard robots.txt extension and support varies by crawler. Use it only for a documented user-agent and only when server logs show that crawl rate is a real problem.
