🤖 Overview

SemrushBot-BA is a dedicated web crawler operated by Semrush, a leading online visibility management and SEO SaaS platform headquartered in Boston, USA. Its primary purpose is to collect publicly available website data exclusively for Semrush’s Backlink Audit tool, which analyzes a site’s backlink profile, identifies toxic links, and monitors link-building campaigns. Unlike the more general SemrushBot (used for Site Audit), SemrushBot-BA focuses on crawling external links and referring domains, feeding data into Semrush’s proprietary backlink index. The bot was first documented in Semrush’s official crawler policy page around 2019 and is regularly updated.

🌐 Technical Behavior

SemrushBot-BA crawls the web by following outbound links discovered on already-indexed pages, targeting both the source and target URLs of backlinks. The crawler uses HTTP/1.1 and HTTP/2 protocols, typically sending requests with a Referer header that indicates the page where the link was found. Request frequency is moderate—Semrush states the bot sends no more than 2 requests per second per IP address, with bursts allowed up to 10 requests per second under their standard rate limit. IP ranges for SemrushBot-BA are published in the SemrushBot IP list on their official help page, which includes addresses from ASN 394639 (Semrush Inc.) and ranges like 185.211.136.0/22, 185.211.137.0/24, and others. The bot resolves to hostnames ending in .crawl.semrush.com. It obeys the standard Crawl-Delay directive in robots.txt and can be throttled via the crawl-delay value.

📋 robots.txt Compliance

SemrushBot-BA fully respects robots.txt directives. According to Semrush’s official documentation at https://www.semrush.com/bot/, the bot checks robots.txt before every crawl and abides by Disallow and Allow rules. Additionally, webmasters can block SemrushBot-BA specifically using the User-agent string SemrushBot-BA in their robots.txt file. Semrush also provides a dedicated opt-out form for site owners who wish to exclude their domain from all Semrush crawling entirely.

🔍 Detection Indicators

The primary User-Agent string is SemrushBot-BA/1.0, sometimes appended with (+http://www.semrush.com/bot.html). The bot also sends the From HTTP header set to [email protected] and a User-Agent that may include a version suffix like SemrushBot-BA/5.2 in newer deployments. Behavioral fingerprints include a consistent pattern of requesting only HTML pages (no images, CSS, or JavaScript), and the bot never accepts cookies or executes JavaScript. It identifies itself via reverse DNS lookups on crawl IPs, which always resolve to *.crawl.semrush.com.

📊 Data Usage

Data collected by SemrushBot-BA is used exclusively within the Semrush Backlink Audit tool to map a domain’s backlink profile, assess link toxicity, and track link growth over time. The bot does not train AI models—Semrush explicitly states that crawling is for SEO analytics, not machine learning. Collected information includes anchor text, link source URLs, HTTP status codes, and page titles of linking pages.

⚙️ Rate Limiting Policy

SemrushBot-BA is rate-limited to prevent overloading web servers; Semrush enforces a default maximum of 2 requests per second per IP and a total daily cap per domain. This policy justifies threshold-based blocking for aggressive crawl sessions, ensuring fair resource usage while still allowing the bot to collect necessary backlink data for legitimate SEO analysis.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.