SemrushBot robots.txt Rules, Blocking, and Crawl Limits

What SemrushBot does

SemrushBot is the main Semrush crawler used for backlink analytics and webgraph data. It is one member of a larger crawler family. Semrush documents separate tokens for products such as Site Audit, Backlink Audit, On Page SEO Checker, Content Toolkit, and other services. Blocking only SemrushBot therefore does not automatically block every Semrush request.

Block the main crawler

To stop the primary SemrushBot from crawling the whole host, use the dedicated group below. This rule controls SemrushBot, not Googlebot or Bingbot, so it does not directly remove pages from ordinary search engines. Use path-specific directives when only selected sections should be excluded.

Control crawl rate

SemrushBot supports the non-standard Crawl-delay directive. Semrush says the crawler accepts intervals up to 10 seconds and reduces larger values to that limit. When no Crawl-delay is present, it adjusts request frequency according to current server load.

Do not use Crawl-delay as a substitute for diagnosing slow pages or server errors. Measure actual request volume first, then choose a reasonable setting if SemrushBot is contributing meaningful load.

Separate Semrush crawler tokens

Semrush publishes individual user-agent tokens including SiteAuditBot, SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, and others. Decide which product traffic you want to control and add each relevant token explicitly. A single main-bot rule cannot express a complete organization-wide block.

Status codes and host scope

The robots.txt response itself matters. Semrush says a 4xx response is treated as if no robots.txt file exists, while a 5xx response prevents SemrushBot from crawling the site. It can follow a 3xx response, but the safest operational setup is a valid file returning HTTP 200 from the correct host root.

Each subdomain needs its own robots.txt file. A rule on example.com does not control shop.example.com. Semrush also notes that it may take up to one hour or 100 requests to discover a changed file.

Verification checklist

When reviewing logs, capture the full user-agent value and requested host. Similar names can belong to different Semrush products, and a rule published on the wrong protocol or subdomain will not apply. Confirm the live robots.txt response before changing firewall or CDN rules.

1. Identify which Semrush crawler token appears in logs.
2. Publish the matching group at the correct host root.
3. Return a valid HTTP 200 response.
4. Add separate rules for relevant subdomains and tools.
5. Allow time for the crawler to detect changes.
6. Validate the final file with the checker.

Use the generator for a broader multi-crawler policy, and review robots.txt HTTP status codes before changing server behavior.