🤖 Overview

Qwantify is the primary web crawler operated by Qwant SAS, the French search engine company based in Paris, to build and refresh its search index. First publicly documented in early 2018, Qwantify is part of Qwant's effort to provide a privacy-focused, European alternative to Google and Bing, with data processing occurring on servers hosted in France. The bot feeds crawled web pages into Qwant's search engine, which serves users across the European Union and Switzerland, emphasizing no user tracking or profiling.

🌐 Technical Behavior

Qwantify crawls the web using a custom scrapy-based engine developed in Python, as referenced in Qwant's open-source repository on GitHub (github.com/Qwant). It performs HTTP/1.1 requests with Accept-Encoding: gzip, deflate and Connection: keep-alive headers. The default crawl frequency is moderate, with a politeness delay of 10 seconds between consecutive requests to the same host, documented in Qwant's robots.txt policy page. IP addresses originate from the AS50474 (Qwant SAS) netblock, primarily 194.150.235.0/24 and 185.233.100.0/22, hosted by OVHcloud and Scaleway. Qwantify supports both HTML and sitemap crawling, with a Max-URLs-per-request limit of 5000 per sitemap.

📋 robots.txt Compliance

According to Qwant's official documentation at help.qwant.com, Qwantify fully honors robots.txt directives, including Disallow, Crawl-delay, and Allow rules. It also respects X-Robots-Tag HTTP headers and noindex meta tags. Site owners can specify a custom crawl delay via Crawl-delay: 10 in their robots.txt, which Qwantify will obey. There are no known documented cases of Qwantify ignoring disallow directives.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/legal/advanced-search/bot). A secondary UA string for mobile content is Mozilla/5.0 (Linux; Android 8.0; SM-G960F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 (compatible; Qwantify/2.0). Behavioral fingerprints include requests from the AS50474 IP range and a mandatory User-Agent header presence; missing or malformed UAs result in immediate 403 response from Qwant's infrastructure. Reverse DNS entries follow the template crawl-*.qwant.com.

📊 Data Usage

Data collected by Qwantify is exclusively used for search indexing and ranking algorithms within Qwant's own search engine. The content is not used to train external AI models, nor sold to third parties, as stated in Qwant's privacy policy at help.qwant.com/legal/privacy. Crawled pages are stored in a NoSQL database on servers within the EU, and indexed content is refreshed periodically based on page change frequency signals from HTTP headers.

⚙️ Rate Limiting Policy

Qwantify is rate-limited by webmasters because its crawling pattern, though compliant, can still generate heavy load on small websites due to concurrent requests across multiple pages. A threshold-based blocking of approximately 100 requests per second per IP is a reasonable policy to protect server resources while still allowing the legitimate crawler access for indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.