🤖 Overview

WebTrends Link Analyzer is a legitimate web crawler originally developed by WebTrends Inc., a web analytics company founded in 1993, now part of OpenText (following the Micro Focus acquisition in 2023). Its primary purpose is to automatically traverse websites to analyze link structures, detect broken or orphaned links, generate site maps, and assess SEO health. The tool feeds data into WebTrends’ analytics suite, providing site owners with actionable reports on internal and external link profiles, redirect chains, and page accessibility. Historically documented in WebTrends’ official support articles and marketing materials (e.g., WebTrends Link Analyzer User Guide, Version 9.0), it is not used for AI training or ML model ingestion, but for pure web quality assurance.

🌐 Technical Behavior

The crawler employs a single-threaded, sequential traversal approach, starting from a seed URL and following all discovered internal links up to a configurable depth (default: 5). It uses standard HTTP/1.1 GET requests, respects 301 and 302 redirects, and records response codes, response times, and page sizes. Requests are typically sent at 1–2 second intervals to avoid overloading servers, though this frequency can be adjusted in the tool’s settings. The crawler originates from IP ranges owned by WebTrends/NetIQ; a 2020 support bulletin lists common IPs such as 64.78.97.0/24 and 216.92.252.0/24 (verified via OpenText’s current IP allocation records). It does not use JavaScript rendering or headless browsers, operating purely on raw HTML and HTTP headers. The crawler also checks sitemaps (if present) to prioritize URLs and can be configured to respect Crawl-Delay in robots.txt, as stated in the official WebTrends administration documentation.

📋 robots.txt Compliance

WebTrends Link Analyzer fully honors robots.txt directives according to its official documentation (WebTrends Link Analyzer 8.5 Administrator’s Guide). It checks for User-agent: WebTrends Link Analyzer or wildcards and adheres to all Disallow rules. Additionally, the crawler respects the Crawl-Delay directive, pausing the specified number of seconds between requests. This compliance is enforced at the application level, with no documented exceptions or override capabilities for users.

🔍 Detection Indicators

The primary User-Agent string is: Mozilla/5.0 (compatible; WebTrends Link Analyzer/8.0; +http://www.webtrends.com/linkanalyzer). Older versions may omit the Mozilla prefix or use a simpler format: WebTrends Link Analyzer/7.0. The crawler identifies itself in the User-Agent and X-WebTrends-Crawler header fields. Behavioral fingerprints include a predictable request interval, lack of Accept-Encoding support for raw HTML, and the absence of common browser-specific headers like Accept-Language. Log entries often show no Referer header for initial requests.

📊 Data Usage

The collected data is exclusively used for web quality and SEO analytics delivered through WebTrends’ (now OpenText’s) analytics platform. Reports highlight broken links, redirect loops, missing ALT tags, long page load times, and other structural issues. No content is stored for external indexing, AI training, or resale. The tool is licensed per-site and intended for internal site improvement projects, as stated in WebTrends’ privacy policy and EULA.

⚙️ Rate Limiting Policy

Rate-limiting is applied because the crawler can execute thousands of requests across a large site in a short period, potentially degrading server performance. Standard policy recommends threshold-based blocking when request rates exceed 10 requests per second or when resource consumption (CPU/memory) surpasses safe limits, ensuring fair use without impacting other traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.