🤖 Overview

amazon-QBusiness is a legitimate web crawler operated by Amazon Web Services (AWS) as part of the Amazon Q Business service, a generative AI assistant announced in November 2023 (released GA in April 2024). Its primary purpose is to crawl and index publicly accessible web content from customer-specified data sources (e.g., internal websites, SharePoint, Confluence, or public URLs) that are configured in Amazon Q Business applications. The collected data feeds the AI model to answer business-specific questions, summarize documents, and generate insights—all within the enterprise’s access controls.

🌐 Technical Behavior

The crawler employs HTTPS GET requests with a respect for standard HTTP redirects (301, 302) and supports conditional GET via ETag and Last-Modified headers to reduce bandwidth. It uses the AWS IP address range published on the official AWS IP address list (https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html), specifically from the AMAZON and AMAZON_CONNECT prefixes. Request frequency is moderate—typically 1–5 requests per second per source—but can scale if multiple data sources are configured. The bot operates with a configurable crawl depth (default 3 levels) and respects robots.txt Crawl-Delay directives if present. It does not execute JavaScript; it only downloads static HTML, PDF, DOCX, TXT, and other supported file types. The crawler identifies itself via the User-Agent header string AmazonQBusiness/1.0 (official documentation from AWS Support, 2024). It also includes a custom header X-Amz-QBusiness-Crawler: true in requests, which can be used for server-side logging. The bot strictly follows index.html root paths and will not follow links to external domains unless the source configuration explicitly allows it.

📋 robots.txt Compliance

Based on official AWS documentation (https://docs.aws.amazon.com/amazonq/latest/userguide/data-source-web-crawler.html), Amazon Q Business fully respects all Disallow directives in robots.txt as well as any Allow overrides. It also honors the Crawl-Delay directive, pausing the specified number of seconds between requests. The crawler will not index any resources blocked by robots.txt, even if a user explicitly adds the URL to the data source. This compliance is enforced at the AWS service level and cannot be disabled by the customer.

🔍 Detection Indicators

Primary identifier: User-Agent string AmazonQBusiness/1.0 (case-sensitive). Requests also include the X-Amz-QBusiness-Crawler header set to true, and the X-Forwarded-For header may contain the AWS internal IP. Behavioral fingerprint: a low request rate (sub‑10 req/s), a narrow crawl depth, and a focus on static content types. The bot will also present a Referer header containing the data source ARN (Amazon Resource Name) in the format arn:aws:qbusiness:region:account-id:datasource/.... These indicators allow system administrators to filter or log the crawler distinctly from other AWS services like Amazon CloudFront or AWS WAF.

📊 Data Usage

Collected data is stored within the customer’s Amazon Q Business index (powered by Amazon OpenSearch Serverless) and used exclusively for AI-powered query responses, document summarization, and content generation within the Q Business application. No data is shared with external parties or used to train foundational models outside the customer’s isolated instance. AWS guarantees that crawled content is encrypted at rest and in transit, and is deleted when the data source is removed (source: AWS GDPR compliance page).

⚙️ Rate Limiting Policy

Rate limiting is recommended for amazon-QBusiness because, although the service adheres to standard politeness, misconfigured sources or large site trees can generate dozens of requests per minute, potentially stressing origin servers. A threshold of 10 requests per second per IP is a safe balance; blocking should only occur if the crawl exceeds 1000 requests per minute from a single AWS IP range, as per typical AWS crawler behavior.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.