🤖 Overview

SemrushBot-SWA is a specialized web crawler operated by Semrush, a leading SaaS platform for SEO, PPC, content marketing, and competitive research. Announced in Semrush’s official crawler documentation at semrush.com/bot/, this bot is specifically designed to perform Site-Wide Audits (SWA)—a feature that analyzes the health, structure, and performance of an entire website at scale. Unlike the main SemrushBot used for indexing and ranking data, SemrushBot-SWA focuses on in-depth technical audits for paying and trial users of Semrush’s Site Audit tool.

🌐 Technical Behavior

SemrushBot-SWA follows a configurable crawl path but typically starts from a user-provided seed URL and recursively discovers internal links up to a depth limit set by the Semrush user. According to Semrush’s official bot page, the crawler respects HTTP status codes (e.g., 3xx redirects, 4xx client errors, 5xx server errors) and reports them in the audit results. It requests pages using HTTP/1.1 and HTTP/2, and the crawl frequency is throttled per domain—usually between 1 and 10 requests per second depending on server response times and the configured crawl budget. The IP ranges used are primarily from Semrush’s owned blocks, including 85.208.96.0/24 and 185.170.60.0/24, as listed on their official IP list published at semrush.com/bot/ip-list/. The crawler identifies itself using the User-Agent string Mozilla/5.0 (compatible; SemrushBot-SWA/1.0; +https://www.semrush.com/bot/) and may also include a custom X-Robots-Tag parsing capability.

📋 robots.txt Compliance

Semrush explicitly states on their bot page that SemrushBot-SWA honors robots.txt directives, including Disallow, Allow, Crawl-Delay, and the nofollow meta tag. The bot also respects X-Robots-Tag HTTP headers when present. However, Semrush notes that the crawler may still visit pages blocked by robots.txt if the user explicitly overrides the crawl settings in the Semrush interface—though this is an edge case. Verified via Semrush’s robots.txt test tool documentation, the bot will skip any path listed as disallowed unless the user forces a re-check.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: Mozilla/5.0 (compatible; SemrushBot-SWA/1.0; +https://www.semrush.com/bot/). Variants include older versions like SemrushBot-SWA/0.9 or an alternative string SemrushBot-SWA/2.0 (if updated). The bot also sets the From header to [email protected] and uses a reverse DNS record under the semrush.com domain. Behavioral fingerprinting reveals a consistent request pattern: it always requests robots.txt first, then follows with HEAD or GET requests, and sends a unique SEMRUSH-BOT-ID cookie for session tracking (noted in Semrush’s changelog on GitHub at github.com/semrush/service-status).

📊 Data Usage

Data collected by SemrushBot-SWA is exclusively used to populate the Site Audit reports inside the Semrush platform. This includes metrics like broken links, page load speed, duplicate content, missing meta tags, and redirect chains. The raw crawl data is stored temporarily on Semrush’s servers and aggregated into user-visible dashboards. Semrush’s privacy policy confirms that no personal information is intentionally harvested; the bot only processes publicly accessible HTML, CSS, and JavaScript resources necessary for technical SEO analysis.

⚙️ Rate Limiting Policy

Although SemrushBot-SWA is a legitimate, non‑malicious crawler, it is rate‑limited by web administrators to prevent excessive load on origin servers. Semrush recommends a default Crawl-Delay of 1 second in robots.txt and warns that sustained high‑frequency requests (above 20 req/s) may trigger automatic throttling from the site side. The rationale is to protect shared hosting environments and maintain fair resource consumption; threshold‑based blocking at 50 requests per minute is a common mitigation strategy recommended in Semrush’s own best‑practice guides.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.