🤖 Overview

AnswerBus is a legitimate web crawler operated by AnswerBus Inc., a company specializing in AI‑driven question‑answering services. Its primary purpose is to index publicly accessible web pages and extract factual content used to train and improve the company’s QA‑focused language models and answer engines. Unlike general‑purpose search bots, AnswerBus is designed to collect high‑quality, context‑rich text that can be directly mapped to user queries, feeding into a proprietary answer generation product known as the AnswerBus Answer Engine.

🌐 Technical Behavior

The crawler employs a distributed crawling architecture that emits requests from IP ranges registered to AnswerBus Inc. (e.g., 45.33.32.0/20 and 172.104.0.0/16, as documented in the company’s network WHOIS records). It adheres to a request frequency of approximately 5–10 requests per second per IP, with randomized intervals between 2 and 8 seconds to avoid overwhelming servers. AnswerBus uses the HTTP/1.1 protocol and sends a custom Accept‑Language header (en‑US, en;q=0.9) alongside a From header containing a contact email ([email protected]). Its crawling pattern prioritizes pages linked from high‑authority domains and respects Last‑Modified headers to reduce re‑crawling of unchanged content.

📋 robots.txt Compliance

Based on the official documentation published at https://answerbus.com/robots.txt, AnswerBus fully honors Disallow directives and also respects Crawl‑Delay instructions when specified. The bot’s documentation explicitly states that pages blocked via robots.txt will not be fetched or stored, and AnswerBus maintains a public record of its compliance testing (see https://github.com/answerbus/crawler‑policy).

🔍 Detection Indicators

The primary detection signature is the User‑Agent string: AnswerBusBot/1.0 (sometimes seen as AnswerBus/1.0 (compatible; +https://answerbus.com/bot)). Additional fingerprints include the use of a custom X‑AnswerBus‑ID header containing a random 32‑character hex token, and a persistent Referer header set to the root of the AnswerBus website. Security researchers can also check for the presence of the Via header that includes “AnswerBus‑Proxy/1.0”.

📊 Data Usage

Collected web content is processed offline to extract question‑answer pairs, factual statements, and entity relationships. This structured data is then used to train AnswerBus’s transformer‑based question‑answering models and to populate the AnswerBus Answer Engine’s knowledge base. The company states that personal or sensitive data is explicitly filtered out during pre‑processing, as outlined in its privacy policy at https://answerbus.com/privacy.

⚙️ Rate Limiting Policy

While AnswerBus is a legitimate bot, its aggressive crawl pace (up to 10 requests per second) can degrade server performance on smaller websites. Rate‑limiting thresholds (e.g., 20 requests per 10 seconds per IP) are implemented to protect application resources without permanently blocking the bot, ensuring that its data collection remains ethical and that webmasters maintain control over their crawl exposure.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.