🤖 Overview

The BacklinksExtendedBot is a legitimate web crawler operated by the SEO analytics platform BacklinksExtended (backlinksextended.com), first documented in 2019. Its primary purpose is to systematically scan websites to collect backlink data, anchor text, and link profiles for the platform’s SEO auditing and competitor analysis tools.

🌐 Technical Behavior

This bot employs a breadth-first crawl strategy, starting from a seed list of URLs submitted by BacklinksExtended users. It follows all discovered outbound links and rel="nofollow" links. Requests are issued over HTTP/1.1 and HTTP/2 with a default interval of 10–15 seconds between successive requests to the same domain. IP addresses originate from a static pool of dedicated servers managed by BacklinksExtended, typically in the 198.51.100.0/24 range (example based on common patterns). The User-Agent header is consistent and includes a version string with a timestamp, e.g., BacklinksExtendedBot/1.2 (+https://backlinksextended.com/bot). The bot does not render JavaScript or execute client‑side scripts; it retrieves only static HTML.

📋 robots.txt Compliance

Per official documentation on the BacklinksExtended website, the bot fully respects robots.txt directives. It checks the file before every crawl session and will refrain from crawling any path marked with Disallow. In practice, webmasters have reported that the bot does not ignore rate‑limiting directives embedded in the Crawl‑Delay field.

🔍 Detection Indicators

The definitive User‑Agent string is Mozilla/5.0 (compatible; BacklinksExtendedBot/1.2; +https://backlinksextended.com/bot). Additional headers include X-BLE-Crawl-ID containing a unique session identifier and From set to [email protected]. The bot’s request pattern shows a fixed interval of 10–15 seconds, no concurrent connections per domain, and a predictable Accept header of text/html,application/xhtml+xml.

📊 Data Usage

Collected backlink data is used exclusively for the BacklinksExtended SaaS dashboard, which provides users with inbound link reports, broken link detection, and competitor backlink comparisons. The platform does not repurpose crawled content for AI model training or advertising; data is retained for a maximum of 90 days before being purged from active indices.

⚙️ Rate Limiting Policy

Rate limiting is recommended because the bot’s persistent scanning of large sites can generate significant server load if left unchecked. A threshold of 300 requests per hour per IP is a common policy to preserve server resources while allowing legitimate SEO audits to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.