🤖 Overview

Zermelo is the official web crawler operated by Qwant, a French search engine that prioritizes user privacy and does not employ tracking cookies or behavioral profiling. Named after the mathematician Ernst Zermelo, this bot is used exclusively to index publicly accessible web content for Qwant’s search results, as documented on Qwant’s official crawler policy page (https://about.qwant.com/legal/crawler-policy/).

🌐 Technical Behavior

Zermelo performs a breadth-first crawl of websites, respecting a default Crawl-delay of 5 seconds unless overridden by the site’s robots.txt. It typically makes requests over HTTP/1.1 and HTTPS, with parallel connections limited to 2 per host, as per Qwant’s published technical specifications. The crawler originates from IP ranges in the 5.39.0.0/16 and 37.59.0.0/16 blocks, allocated to Qwant’s infrastructure provider. Qwant periodically updates these ranges in a dedicated text file at https://about.qwant.com/crawler-ips.txt. The bot also advertises support for the If-Modified-Since header to reduce bandwidth usage, but does not send cookies or JavaScript execution requests.

📋 robots.txt Compliance

According to Qwant’s official crawler documentation, Zermelo fully honors Disallow and Allow directives in robots.txt, including wildcard patterns. It also supports the Crawl-delay directive, allowing webmasters to throttle its request rate. Qwant explicitly advises that blocking Zermelo via robots.txt will exclude the site from Qwant’s search index.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Zermelo/1.0; +https://about.qwant.com/legal/crawler-policy/). A secondary variant Qwantify/1.0 may also appear for specific use cases like freshness checks. The crawler identifies itself with a From header set to [email protected] and always includes a User-Agent token matching the pattern above. Reverse DNS lookups on request IPs resolve to hostnames ending in .qwant.com.

📊 Data Usage

Collected web pages are processed and stored exclusively for Qwant’s search indexing service. Qwant states in its privacy policy that it does not use crawled content for AI model training or any secondary purpose, distinguishing it from many other search engine operators. Indexed data is refreshed periodically to maintain result freshness.

⚙️ Rate Limiting Policy

Rate limiting Zermelo is recommended only if its request volume interferes with normal site operations, as Qwant limits each crawler instance to 2 concurrent connections per host. A threshold-based block may be implemented after sustained rates exceed 10 requests per minute per IP, as advised by Qwant’s own best-practice guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.