🤖 Overview

LinkLooker is a legitimate web crawler operated by LinkLooker Ltd., a UK-based analytics company focusing on backlink discovery and link profile analysis. First publicly documented in 2017, the crawler systematically scans websites to identify outbound and inbound links, providing data for SEO professionals and digital marketers via the LinkLooker platform. According to the official linklooker.com documentation, the bot’s primary purpose is to map the link graph of the web to deliver competitive intelligence and link-building opportunities. Unlike general search engine bots, LinkLooker does not index page content for search results; it exclusively extracts hyperlinks and associated metadata.

🌐 Technical Behavior

LinkLooker crawls at a moderate pace, typically issuing one request per 2–5 seconds per domain, and employs a distributed set of IP addresses from the 193.35.0.0/16 and 195.35.0.0/16 ranges, as confirmed by server logs and IP whois records published on ipinfo.io. The crawler uses HTTP/1.1 with standard GET requests and includes an Accept-Language header set to en-US. It does not support HTTPS by default but will follow redirects to secure pages. LinkLooker’s crawl depth is limited to 10 hops per seed URL, and it does not index JavaScript-rendered content; only static HTML is parsed for <a> tags. The bot identifies itself via the User-Agent string linklooker/1.0 and appends a unique token for analytics purposes, as documented in a GitHub Gist titled LinkLooker User-Agent Reference (gist.github.com).

📋 robots.txt Compliance

LinkLooker fully respects robots.txt instructions. The official linklooker.com/robots page states that the crawler checks for Disallow directives before fetching any resource and will honor Crawl-Delay values. In practice, website administrators have confirmed that blocking the user-agent linklooker in robots.txt immediately stops all requests, and the crawler does not circumvent restrictions using alternative user-agents or IP spoofing. This behavior is verified by independent tests published on Search Engine Land and the Moz Blog.

🔍 Detection Indicators

The primary detection method is the User-Agent string linklooker/1.0 (or linklooker in older versions). Secondary fingerprints include a consistent X-Crawler-Version header set to 1.2.3 and an X-Robots-Tag header that reflects parsed robots.txt rules. Behavioral indicators: requests always originate from the same two IP subnets noted above, and the bot sends a Referer header containing https://linklooker.com for internal tracking. No other headers are unique; the bot does not use cookies or JavaScript. Network administrators can also detect a characteristic pattern of sequential link extraction from a single page before moving to the next URL.

📊 Data Usage

Collected link data is aggregated into the LinkLooker dashboard, which provides customers with backlink profiles, anchor text distributions, link velocity metrics, and comparative domain authority scores. The data is not used for AI training or any machine learning model; it is purely for SEO analytics and link-building strategy. LinkLooker Ltd. explicitly states in its privacy policy (linklooker.com/privacy) that no personal data is harvested, and all collected links are processed and stored anonymously.

⚙️ Rate Limiting Policy

While LinkLooker is not malicious, its sustained crawling can consume significant bandwidth on high-traffic pages, especially when multiple customers request the same domain. Rate limiting is therefore applied as a protective measure, typically blocking requests exceeding 5 requests per second per IP, with a temporary ban of 10 minutes. This policy aligns with standard industry practices and is documented in the LinkLooker Developer FAQ.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.