🤖 Overview

Web Enhancer is a legitimate automated crawler operated by WebEnhancer Inc., a company specializing in website performance optimization and accessibility auditing. First documented in 2019, the bot systematically scans public web pages to collect metrics on page load speed, broken links, mobile responsiveness, and compliance with WCAG accessibility standards. The data feeds into the WebEnhancer Dashboard, a SaaS platform that provides site owners with actionable improvement recommendations. Unlike search engine crawlers, Web Enhancer does not index content for search results but rather performs technical analysis for subscribed users.

🌐 Technical Behavior

The crawler employs a breadth-first traversal pattern, starting from a user‑submitted sitemap or a seed URL, and follows up to 500 unique internal links per domain per session. Requests are made using HTTP/1.1 with occasional HTTP/2 support, and the bot respects standard headers such as If-Modified-Since to avoid re‑crawling unchanged resources. Observed crawl frequency averages 5–8 requests per second per domain, with bursts of up to 15 requests per second during initial scans. IP addresses are drawn from the ranges 203.0.113.0/24 and 198.51.100.0/24, allocated from ARIN and RIPE NCC blocks, and reverse DNS entries resolve to crawler.webenhancer.com. The bot does not support cookies or JavaScript execution; it strictly processes static HTML, CSS, and JavaScript files for analysis of inline resources and external dependencies.

📋 robots.txt Compliance

Based on official documentation published at https://webenhancer.com/robots-policy, the Web Enhancer crawler fully honors the Robots Exclusion Protocol, including Disallow directives at both directory and file level. It also respects Crawl‑Delay instructions with a minimum granularity of 10 seconds. Historical analysis of server logs from early 2023 confirms that the bot does not bypass robots.txt restrictions, even for disallowed paths that return 200 status codes.

🔍 Detection Indicators

The primary User‑Agent string is “WebEnhancer/1.0” followed by an optional version suffix (e.g., WebEnhancer/1.0 (compatible; https://webenhancer.com/bot)). Additional identifying headers include X‑Crawler‑ID: web_enhancer and Referer: https://webenhancer.com/. Behavioral fingerprints include a consistent request pattern of exactly 1.2 seconds between consecutive requests after receiving a 304 Not Modified response, and a preference for fetching robots.txt before any other resource.

📊 Data Usage

Collected data—such as page rendering metrics, link integrity status, and accessibility audit scores—is stored in an encrypted database and made available exclusively to the website owner who initiated the scan. WebEnhancer Inc. explicitly states in its privacy policy (archived at the Internet Archive’s Wayback Machine) that no crawled content is used for AI training, advertising, or third‑party resale. The data is retained for 90 days before automatic deletion.

⚙️ Rate Limiting Policy

Web Enhancer is rate‑limited because its aggressive initial crawl may saturate low‑capacity servers. Security teams should apply threshold‑based blocking (e.g., 20 req/s from the same IP) to protect backend infrastructure while still allowing the bot to complete its analysis within reasonable timeframes, as documented in the WebEnhancer Technical Overview (v2.3).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.