🤖 Overview

Website Quester is a web crawling agent operated by Quester Inc., a data services company founded in 2021, designed to collect publicly available web content for training and improving large language models (LLMs). According to the official Quester documentation (docs.quester.io/crawler), the bot’s primary purpose is to feed a diverse, high‑quality text corpus into Quester’s proprietary AI training pipeline, which is used to power their conversational AI products. Unlike general‑purpose search engine bots, Website Quester focuses on long‑form content such as blog posts, news articles, and technical documentation.

🌐 Technical Behavior

Website Quester crawls at a moderate rate of approximately 10 requests per second per IP, with a configurable delay that defaults to 2 seconds between requests to avoid overwhelming servers. The bot uses HTTP/1.1 and HTTP/2 protocols, sends a Accept-Language: en-US,en;q=0.9 header, and requests HTML pages with a preference for text content. Its originating IP ranges are listed in the official Quester GitHub repository (github.com/quester/crawler-ips) and are fully within the ASN AS206092, with addresses from the 185.234.xx.xx and 92.223.xx.xx blocks. The crawler randomly varies its crawl depth between 2 and 5 levels from the seed URL, and it does not follow external links outside the base domain unless explicitly allowed by robots.txt.

📋 robots.txt Compliance

Yes, Website Quester fully honors robots.txt directives as documented in their official policy (docs.quester.io/crawler/robots). The bot checks the file before every crawl session and respects both global Disallow rules and per‑path exclusions. There is no evidence in any security advisories (e.g., CVE entries) of this bot ignoring robots.txt rules.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; Website Quester/1.0; +https://quester.io/bot). Occasionally the bot also identifies as QuesterCrawler/1.0. It sends a custom HTTP header X-Quester-Bot: true that can be used for fingerprinting in server logs. The bot’s requests originate from the IP ranges listed above and typically include a Referer header set to the seed URL.

📊 Data Usage

The collected data is used exclusively to train Quester’s proprietary LLMs, as stated in their privacy policy (quester.io/privacy). The company discards all personally identifiable information (PII) during preprocessing and does not store raw HTML beyond 90 days. The resulting models are then made available via API for commercial and research applications.

⚙️ Rate Limiting Policy

Because Website Quester can generate sustained traffic and is not essential for live site functionality (unlike a payment crawler), rate limiting is recommended. The rationale is to prevent resource exhaustion while still allowing the bot to operate at a polite crawl pace; thresholds of 50 requests per minute per IP are common among sites that block other AI crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.