🤖 Overview

Obvius External Linkcheck is a web crawler operated by Obvius, a company offering website monitoring and SEO tools, specifically designed to scan websites for broken or invalid external hyperlinks. Its sole purpose is to help webmasters maintain link integrity by providing detailed reports on the status of outbound links, thereby improving user experience and search engine optimization.

🌐 Technical Behavior

The bot requests individual pages from a target website and parses the HTML to extract all href attributes pointing to external domains. It then validates each external link by issuing HTTP HEAD or GET requests to the destination, checking for status codes such as 200 (success), 301/302 (redirect), 404 (not found), or 500 (server error). According to Obvius documentation and public webmaster observations, the crawler operates at a moderate rate — typically one request every few seconds per host — to avoid overwhelming servers. The IP addresses originate from Obvius’s own infrastructure, often identifiable via reverse DNS lookups to obvius.com. The bot does not follow internal links beyond the initial page; it focuses exclusively on external links present on crawled pages, making it a focused, non-indexing crawler.

📋 robots.txt Compliance

Obvius explicitly states that their link checker respects robots.txt directives. If a path is disallowed, the bot will not fetch pages from that directory, though it may still check external links found on allowed pages. Evidence from webmaster forums and Obvius’s own support pages confirms that the bot adheres to Disallow rules and can be blocked entirely by adding a corresponding user-agent line in robots.txt.

🔍 Detection Indicators

The primary identifier is the User-Agent string: Obvius External Linkcheck or sometimes Mozilla/5.0 (compatible; Obvius External Linkcheck; +http://www.obvius.com). Requests typically lack browser-like headers such as Accept-Language or Referer, and exhibit a consistent, periodic request interval. The IP range can be traced to Obvius’s registered ASN, which is publicly listed in WHOIS records, and the bot often includes a contact URL in its User-Agent for identification.

📊 Data Usage

The collected data — specifically the HTTP status codes, response times, and redirect chains of external links — is aggregated into a report for the subscribing website owner. No content from the pages themselves is stored or used for AI training or search indexing; only link metadata is retained temporarily for reporting purposes. This service helps webmasters detect and fix broken links, thereby maintaining site credibility and SEO ranking.

⚙️ Rate Limiting Policy

While Obvius’s bot is designed with built-in rate limiting to be polite, web administrators may still choose to rate-limit or block it if the crawl frequency exceeds their server capacity. There is no requirement to allow the bot for search engine functionality; blocking it only prevents link checking for that domain, making it a low-impact target for rate limiting.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.