SemrushBot is Semrush’s crawler for backlink and webgraph data. You can block it with a SemrushBot group or use Crawl-delay up to Semrush’s documented 10-second limit, but other Semrush tools use separate crawler tokens.
User-agent: SemrushBot Disallow: /
SemrushBot is the main Semrush crawler used for backlink analytics and webgraph data. It is one member of a larger crawler family. Semrush documents separate tokens for products such as Site Audit, Backlink Audit, On Page SEO Checker, Content Toolkit, and other services. Blocking only SemrushBot therefore does not automatically block every Semrush request.
To stop the primary SemrushBot from crawling the whole host, use the dedicated group below. This rule controls SemrushBot, not Googlebot or Bingbot, so it does not directly remove pages from ordinary search engines. Use path-specific directives when only selected sections should be excluded.
SemrushBot supports the non-standard Crawl-delay directive. Semrush says the crawler accepts intervals up to 10 seconds and reduces larger values to that limit. When no Crawl-delay is present, it adjusts request frequency according to current server load.
Do not use Crawl-delay as a substitute for diagnosing slow pages or server errors. Measure actual request volume first, then choose a reasonable setting if SemrushBot is contributing meaningful load.
Semrush publishes individual user-agent tokens including SiteAuditBot, SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, and others. Decide which product traffic you want to control and add each relevant token explicitly. A single main-bot rule cannot express a complete organization-wide block.
The robots.txt response itself matters. Semrush says a 4xx response is treated as if no robots.txt file exists, while a 5xx response prevents SemrushBot from crawling the site. It can follow a 3xx response, but the safest operational setup is a valid file returning HTTP 200 from the correct host root.
Each subdomain needs its own robots.txt file. A rule on example.com does not control shop.example.com. Semrush also notes that it may take up to one hour or 100 requests to discover a changed file.
When reviewing logs, capture the full user-agent value and requested host. Similar names can belong to different Semrush products, and a rule published on the wrong protocol or subdomain will not apply. Confirm the live robots.txt response before changing firewall or CDN rules.
Use the generator for a broader multi-crawler policy, and review robots.txt HTTP status codes before changing server behavior.
Semrush documents the main SemrushBot as a crawler for backlink analytics and webgraph data.
No. Semrush uses separate tokens such as SiteAuditBot, SemrushBot-BA, and SemrushBot-SI for different products.
Semrush says SemrushBot accepts intervals up to 10 seconds; larger values are reduced to that limit.
Semrush says a 4xx response is treated as though no robots.txt file exists, so the crawler assumes there are no restrictions.
Semrush says discovery can take up to one hour or 100 requests after the file changes.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
Common robots.txt Mistakes and How to Fix ThemThe most damaging robots.txt mistakes are publishing the file on the wrong host, blocking every crawler with a wildcard, confusing crawl control with deindexing or security, using incorrect path rules, and failing to test the complete file. Validate exact URLs and crawler groups before deployment.
How to Check Whether AI Crawlers Are BlockedFetch the live robots.txt file, select the crawler’s exact product token, and test the specific URL against the matching group and longest Allow or Disallow rule. Then check HTTP responses, caching, firewall rules, and logs, because a crawler may be allowed by robots.txt yet blocked by the server—or disallowed by robots.txt while an unidentified scraper still reaches the page.
robots.txt HTTP Status Codes: 200, 3xx, 4xx, and 5xxA successful 2xx response lets crawlers parse robots.txt. Redirects may be followed, ordinary 4xx responses often mean no usable robots.txt restrictions, while 5xx and network failures are treated as temporary retrieval problems. Exact behavior and cache timing vary by crawler, so return a stable 200 response for a valid file and never use 401 or 403 as a substitute for crawl rules.
Crawl-delay in robots.txt: Meaning and Crawler SupportCrawl-delay asks a crawler to wait between requests, but it is a non-standard robots.txt extension and support varies by crawler. Use it only for a documented user-agent and only when server logs show that crawl rate is a real problem.
