🤖 Overview

Linkdex (operated by Linkdex Limited, a UK-based SEO and digital marketing company founded in 2004) is a legitimate search engine crawler known as LinkdexBot. Its primary purpose is to index web pages for the Linkdex Search platform and to provide SEO analytics data—including backlink profiles, indexing status, and site health metrics—to paying subscribers of its SEO suite. The bot feeds into Linkdex’s proprietary search index and its SEO tools, which were historically used by agencies and enterprises for competitive intelligence.

🌐 Technical Behavior

LinkdexBot performs HTTP/1.1 GET requests and respects standard crawling etiquette. It typically requests robots.txt before crawling a site and obeys Crawl-delay directives if set. Its request frequency is moderate, often sending 1–2 requests per second per host under normal operation, though this can scale up during deep crawls of large domains. The bot identifies itself via a User-Agent string containing linkdexbot (e.g., Mozilla/5.0 (compatible; linkdexbot/3.1; +http://www.linkdex.com/about/bots/)) and originates from IP ranges belonging to Linkdex’s cloud infrastructure (historically AWS and dedicated servers in the UK). It fetches both HTML and CSS/JavaScript resources to understand page structure, but it does not execute JavaScript for content extraction.

📋 robots.txt Compliance

Official documentation from Linkdex (archived at http://www.linkdex.com/about/bots/) states that the crawler fully honors Disallow directives in robots.txt. The bot also respects noindex meta tags and X-Robots-Tag HTTP headers. No public reports exist of LinkdexBot ignoring robots.txt instructions; it is considered a well-behaved, standards-compliant crawler.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string containing linkdexbot (versions 1.x through 4.x have been observed). Additional fingerprints include a consistent X-Forwarded-For header set to its own IP, and a Referer header sometimes set to http://www.linkdex.com/bots/. The bot does not employ evasive tactics—its IPs are listed in public reverse DNS records (e.g., bot-.linkdex.com).

📊 Data Usage

Collected data is used for SEO analytics and search indexing. Linkdex processes crawled content to build backlink graphs, compute domain authority scores, and provide keyword ranking data. Historical announcements (e.g., on Search Engine Land, 2012) confirm that Linkdex also aggregated data for its proprietary Linkdex Visibility Score, which measured a site’s overall search engine performance. The data is not used for AI model training but purely for SEO tooling and index updates.

⚙️ Rate Limiting Policy

Although LinkdexBot is legitimate and respects automated signals, security policies often rate-limit it to prevent excessive load on origin servers. The recommended approach is to set a Crawl-delay: 10 in robots.txt or implement threshold-based blocking (e.g., >100 requests per minute) to protect API endpoints or dynamic content while still allowing the bot to index static pages.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.