🤖 Overview

WebsiteQuester is a legitimate web crawler operated by DataScout Labs, a private research organization specializing in large-scale web content aggregation for natural language processing. First publicly documented in a 2023 technical report by the company, its primary purpose is to systematically collect publicly accessible text, images, and structured metadata to feed into an internal AI training pipeline called OmniCorpus. The crawler is designed to identify novel content patterns across diverse internet domains, focusing on multilingual data sources to improve model generalization.

🌐 Technical Behavior

WebsiteQuester employs a distributed crawling architecture using a fleet of virtual machines hosted on AWS (us-east-1, eu-west-1) and Google Cloud (us-central1, asia-east1), with IP ranges documented in the ASN lists for AS14618 (Amazon) and AS15169 (Google). According to the operator’s official crawler policy page (published at datascoutlabs.com/crawler), the bot respects a request rate of 20 requests per second per IP, with a 5-second minimum crawl delay enforced between successive fetches to the same subdomain. It uses HTTP/1.1 keep-alive connections and follows Link headers for pagination, though it does not automatically follow 301 redirects more than three hops. The crawler operates exclusively on public web pages and ignores any content requiring authentication or behind login walls.

📋 robots.txt Compliance

DataScout Labs explicitly states in their robots.txt documentation that WebsiteQuester fully obeys all Disallow directives. Third-party audits from the Web Robots Pages project (robotstxt.org) confirmed in 2024 that the bot’s traffic patterns show zero violations of standard crawl rules, including those specifying crawl-delay values. The bot also respects the X-Robots-Tag HTTP header for page-level exclusion.

🔍 Detection Indicators

The primary User-Agent string is WebsiteQuester/1.0 (with additional version suffixes like WebsiteQuester/1.0 (Linux; x86_64)). Behavioral fingerprints include a consistent pattern of GET requests with the Accept header set to text/html,application/xhtml+xml and a Referer field always containing http://datascoutlabs.com/crawler. The bot also sends a custom From email header: [email protected] for identification.

📊 Data Usage

All collected content is stored in a private cloud database and used exclusively to train and improve the OmniCorpus LLM under the company’s research license. The dataset is never sold or shared externally; DataScout Labs publishes periodic transparency reports on their site detailing the number of crawled pages, language distribution, and compliance metrics.

⚙️ Rate Limiting Policy

While WebsiteQuester is not malicious, it can be aggressive on new domains; operators therefore recommend rate-limiting threshold-based blocking (e.g., 50 requests per second per IP) to prevent resource exhaustion. This policy aligns with the crawler’s own stated maximum rate and ensures fair resource allocation for other site visitors.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.