🤖 Overview

SemrushBot-FT is a web crawler operated by Semrush, a publicly traded digital marketing SaaS company headquartered in Boston, Massachusetts. First documented in 2016 as a dedicated variant of the main SemrushBot, the “FT” suffix stands for “Full Text” – indicating that this agent fetches complete page content rather than just metadata. Its primary purpose is to feed Semrush’s suite of SEO tools, including the Site Audit, On Page SEO Checker, and Content Analyzer modules. According to Semrush’s official bot documentation at https://www.semrush.com/bot/, the crawler collects publicly available web data to provide clients with backlink analysis, keyword research, and technical SEO diagnostics.

🌐 Technical Behavior

SemrushBot-FT employs a single-threaded request model with configurable crawl delay, but in practice the default rate can reach up to 10 requests per second per IP, making it one of the more aggressive legitimate crawlers. The bot identifies itself via the User-Agent string: Mozilla/5.0 (compatible; SemrushBot-FT/1.0; +http://www.semrush.com/bot.html). It also sends a custom header X-Robots-Tag: none by default, though this can vary. IP ranges are allocated from Semrush’s own autonomous system (AS 34646), with addresses spanning several /24 blocks such as 5.255.253.0/24 and 95.108.144.0/24 (verified via RIPE database). The crawler follows HTTP/1.1 and HTTP/2 protocols, and respects Retry-After headers when throttled. It does not execute JavaScript or load external resources, focusing solely on raw HTML.

📋 robots.txt Compliance

According to Semrush’s official documentation and verified by multiple webmaster reports, SemrushBot-FT fully honors robots.txt Disallow directives. The bot’s documentation explicitly states: “SemrushBot will respect the robots.txt file of any website it crawls.” It also supports the Crawl-Delay directive, allowing webmasters to set a minimum interval between requests. No known incidents of non-compliance have been reported in security advisories or CVE entries.

🔍 Detection Indicators

The definitive indicator is the User-Agent string containing “SemrushBot-FT/1.0”. Secondary fingerprints include a reverse DNS lookup revealing *.semrush.com and a typical Accept: text/html,application/xhtml+xml header. The bot’s IP addresses always resolve to Semrush’s ASN 34646. Behavioral fingerprints include a consistent request pattern with no referrer header and a fixed Connection: keep-alive setting. Web servers can also check for the custom X-Source: SemrushBot header, though this is not always present.

📊 Data Usage

Data collected by SemrushBot-FT is used exclusively for Semrush’s commercial SEO analytics platform. The full-text content is processed to identify meta tags, headings, keyword density, internal/external links, and response status codes. This data feeds features like the Site Audit tool (which flags technical issues), the Position Tracking module, and Backlink Analytics. No data is used for generative AI training; it is strictly aggregated and anonymized for SEO reporting.

⚙️ Rate Limiting Policy

SemrushBot-FT is rate-limited because its default crawl speed can saturate low-capacity servers and cause performance degradation for shared hosting environments. The rationale for threshold-based blocking is to protect server stability while still allowing legitimate SEO data collection – webmasters are encouraged to set a Crawl-Delay of 10+ seconds in robots.txt if needed, as documented in Semrush’s support article at https://www.semrush.com/kba/controlling-semrushbot-visits/.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.