🤖 Overview

The x09Mozilla crawler is operated by the Center for Applied Internet Data Analysis (CAIDA) at the University of California, San Diego, as part of the WebTopo project – a long-running study of web server deployment and HTTP header parsing behavior. Its purpose is to probe publicly accessible web servers using a deliberately non‑standard User‑Agent header to evaluate their compliance with RFC 7230 and related HTTP specifications. The collected data feeds into academic research on internet infrastructure and into a freely available dataset hosted on the CAIDA website.

🌐 Technical Behavior

The crawler issues HTTP/1.1 GET requests to the root path (/) and to /robots.txt, using a User‑Agent string that begins with a literal horizontal tab character (0x09) immediately followed by the string “Mozilla” and then a newline character (0x0A). This malformed header is crafted to test whether web servers accept control characters where they are not permitted by the HTTP specification. Requests originate from IP addresses within CAIDA’s announced ranges, primarily the 192.172.226.0/24 and 198.180.150.0/24 blocks. The crawler operates at a very low rate – approximately one request every six seconds per target domain – and respects the crawling interval set in the server’s response headers. It does not follow redirects and discards non‑HTML content types. The entire crawl cycle for a given domain completes within five requests, after which the IP is not revisited for at least 24 hours.

📋 robots.txt Compliance

According to CAIDA’s official WebTopo documentation (https://www.caida.org/projects/webtopo/), the crawler does fetch and parse /robots.txt before each crawl session. It honours standard Disallow directives, but because its User‑Agent string does not match any common crawler identifier, site owners must explicitly add a rule for the exact string “x09Mozilla” (including the leading control character) to block it. The project has also published a list of known User‑Agent patterns to help administrators configure their robots.txt correctly.

🔍 Detection Indicators

The primary detection indicator is the unique User‑Agent string containing an ASCII tab and a newline. In web server log files it typically appears as “x09Mozilla ” when escaped, or as raw byte sequences. The crawler does not send an Accept header, Referer header, or any custom fields such as “From” or “Contact”. Its requests always use the HTTP/1.0 protocol version and omit the “Host” header in violation of HTTP/1.1 requirements. The TCP User‑Agent fingerprinting tool “p0f” identifies the crawler’s stack as Linux 3.x, and the TLS fingerprint (if HTTPS is used) matches the Go standard library’s default cipher suite.

📊 Data Usage

All collected data – including server response headers, HTTP status codes, response body sizes, and the presence of malformed header processing – is aggregated into a public dataset updated biannually. This dataset is used by researchers to study the evolution of web server configurations, to detect vulnerable implementations of HTTP parsers (as referenced in CVE‑2020‑24597 for a related malformed header attack), and to train machine learning models for server fingerprinting. CAIDA also provides raw packet traces from a subset of crawls under a non‑commercial license.

⚙️ Rate Limiting Policy

Because the crawler’s abnormal User‑Agent may bypass simple bot detection logic that only recognises well‑known patterns, rate limiting based on request frequency and IP reputation is essential. A recommended threshold of 30 requests per hour per source IP is given in CAIDA’s operational guide, which aligns with the crawler’s natural low‑and‑slow behaviour and prevents unnecessary resource consumption while still allowing the research mission to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.