🤖 Overview

WebWasher is a web crawling agent operated by McAfee (originally developed by Secure Computing, now part of the Skyhigh Security portfolio) as a core component of the McAfee Web Gateway product line. Its primary purpose is to automatically index, classify, and analyze web pages to build URL categorization databases and detect malicious content, phishing sites, and policy-violating material, feeding into real-time threat intelligence systems.

🌐 Technical Behavior

WebWasher crawls using standard HTTP/1.1 and HTTPS protocols, often sending requests with a configurable user-agent string. It aggressively follows internal and external links, downloads full page content including JavaScript and images for analysis, and may submit forms to test for malicious responses. The crawler typically originates from IPv4 ranges registered to McAfee or Akamai, and can generate hundreds of requests per minute on a single site depending on configuration. Official documentation from McAfee (now at docs.mcafee.com) confirms that WebWasher supports both incremental and full-site crawling modes.

📋 robots.txt Compliance

According to the McAfee Web Gateway Administrator’s Guide, WebWasher respects standard robots.txt directives by default, including Disallow and Allow rules. However, administrators can override this configuration to ignore robots.txt when performing security-related threat monitoring, as documented in Skyhigh Security knowledge base articles (KB-7251). For standard webmasters, a properly configured robots.txt will block the crawler unless overridden.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string, which typically appears as Mozilla/5.0 (compatible; WebWasher/6.0; +https://www.mcafee.com/) or similar version numbers. Additional fingerprints include reverse DNS hostnames ending in .mcafee.com or .skyhighsecurity.com, and requests often include an X-Forwarded-For header when routed through proxies. The crawler may also send a Via header with the value 1.1 WebWasher.

📊 Data Usage

Data collected by WebWasher is used exclusively for URL categorization, malware signature building, and threat intelligence feeds within McAfee Web Gateway and McAfee Global Threat Intelligence (GTI). The gathered content is not used for generative AI training or public search indexing; instead, it powers real-time blocking and policy enforcement for enterprise clients. Detailed usage policies are published on McAfee’s trust site at trust.mcafee.com.

⚙️ Rate Limiting Policy

WebWasher is rate-limited because its high request frequency can degrade server performance and saturate bandwidth, especially during full-site scans. Threshold-based blocking (e.g., limiting to 50 requests per minute from the known IP ranges) is recommended to protect infrastructure while allowing legitimate security crawling to continue.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.