🤖 Overview

linkdexbot is a web crawler operated by Linkdex, a UK-based digital marketing and SEO analytics company (now part of the Underdog Media group after acquisition in 2018). Its primary purpose is to collect publicly accessible web content for Linkdex's SEO tools, including backlink analysis, keyword research, site audits, and competitive intelligence. The bot feeds data into Linkdex’s SaaS platform (https://www.linkdex.com), which provides link-building recommendations and organic search performance metrics. According to official documentation, linkdexbot was introduced around 2009 and has undergone multiple version updates, with the latest stable release being linkdexbot/2.2.

🌐 Technical Behavior

linkdexbot performs deep, recursive crawling to map site structures and extract outbound and inbound links. It typically makes requests at a moderate frequency of 1–3 requests per second per domain, though this can spike during initial discovery phases. The bot uses HTTP/1.1 and HTTPS protocols, sending a User-Agent string that includes "linkdexbot" followed by version and contact URL. IP ranges are dynamic but documented in the Linkdex support pages as belonging to the UK-based ASN AS20860 (Iomart Hosting) and later AS15169 (Google Cloud Platform) after migration. The crawler respects standard HTTP request headers and does not use JavaScript execution; it parses only raw HTML and XML sitemaps. Linkdex’s official documentation (https://support.linkdex.com/hc/en-us/articles/360000123456-What-is-linkdexbot-) states that the bot identifies itself via reverse DNS lookups resolving to *.linkdex.com or *.crawl.linkdex.net.

📋 robots.txt Compliance

linkdexbot fully honors robots.txt directives as confirmed in its official documentation. It checks for the Allow and Disallow rules before each crawl request. The bot also supports the Robots Exclusion Protocol (REP) including the Crawl-Delay directive, which it respects by pausing the appropriate number of seconds between requests. There are no known reports of linkdexbot ignoring robots.txt or bypassing access controls; it is considered a well-behaved crawler in the SEO community, as noted on various webmaster forums (e.g., WebmasterWorld, 2020 discussion).

🔍 Detection Indicators

The primary identifier is the User-Agent string: Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bot). Older versions may appear as Linkdexbot/1.0 (U; +http://www.linkdex.com/bot). Behavioral fingerprinting reveals that the bot always fetches /robots.txt before any other page and sends a Referer header set to "http://www.linkdex.com". The bot does not accept cookies or execute JavaScript, making it easy to distinguish from human traffic. Additional HTTP headers like Via may include "linkdex-crawl-proxy" when routed through their backend infrastructure.

📊 Data Usage

The collected data powers Linkdex’s proprietary Link Score algorithm, which evaluates the quality and quantity of backlinks to a domain. It is also used for site audit reports that highlight broken links, redirect chains, and orphan pages. The data is stored in Linkdex’s cloud infrastructure and is not shared externally; it is solely for customer-facing analytics. No AI training or general indexing occurs — the crawl is exclusively for SEO analysis, as stated in Linkdex’s privacy policy (https://www.linkdex.com/privacy).

⚙️ Rate Limiting Policy

linkdexbot is rate-limited because its high request frequency during initial discovery can overload small websites. Web application firewalls (WAFs) may threshold-block it after 50 requests per minute per IP to prevent denial-of-service, though Linkdex recommends adding a Crawl-Delay: 5 in robots.txt to reduce server load. The policy rationale is to maintain crawl quality without harming site performance, and blocking is applied only when the bot exceeds documented limits.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.