🤖 Overview

WebMoose is a legitimate web crawler operated by the Mozilla Foundation, first publicly documented in 2022, designed to collect public web content for the purpose of testing web security and compatibility features within the Firefox browser ecosystem.

🌐 Technical Behavior

WebMoose typically crawls at a rate of 1–3 requests per second per origin, using HTTP/1.1 and HTTP/2 protocols. It respects Cache-Control headers and requests GET only. Its IP ranges are drawn from Mozilla’s ASN (AS54113) and include addresses in the 63.245.x.x and 104.16.x.x blocks. The bot does not follow JavaScript redirects and only crawls pages with HTML content.

📋 robots.txt Compliance

WebMoose honors Disallow directives as documented on the official Mozilla crawler policy page (available at wiki.mozilla.org/WebMoose). It caches robots.txt for up to 24 hours and will not crawl paths listed as disallowed.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; WebMoose/1.0; +https://webmoose.mozilla.org/bot). Behavioral fingerprints include a consistent request interval of 1–3 seconds and the absence of Accept-Encoding: gzip on initial requests. The bot sends a custom X-WebMoose-Version header with each request.

📊 Data Usage

Collected data is used exclusively for Mozilla’s internal web compatibility testing, security vulnerability scanning, and improving the Firefox rendering engine. No data is sold or used for AI training; results are aggregated into public bug reports on Bugzilla.

⚙️ Rate Limiting Policy

WebMoose is rate‑limited because its high crawl volume can consume server resources if left unrestricted, and threshold‑based blocking ensures fair access for all legitimate bots without impacting user traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.