🤖 Overview

bedrockbot is a web crawler operated by Amazon Web Services (AWS) as part of the Amazon Bedrock service, first documented in early 2024. Its primary purpose is to collect publicly available web content to train and improve foundation models offered through Bedrock, such as Amazon Titan, Claude, and other third-party models. The bot is explicitly designed to support responsible AI development by respecting website owners' preferences.

🌐 Technical Behavior

According to AWS documentation (https://docs.aws.amazon.com/bedrock/latest/userguide/crawlers.html), bedrockbot crawls using HTTP/1.1 and HTTP/2 protocols, making GET and HEAD requests. It typically requests pages at a rate of 1 request per second per IP, with bursts up to 5 requests per second. The crawler uses IP addresses from AWS's own EC2 and CloudFront ranges, which are publicly listed in the AWS IP address ranges JSON (https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html). Crawl frequency is configurable via the Amazon Bedrock console, and the bot respects Crawl-Delay directives in robots.txt with a default delay of 10 seconds. It does not fetch binary files (e.g., images, PDFs) by default.

📋 robots.txt Compliance

bedrockbot honors Disallow directives in robots.txt files, as confirmed in the official AWS crawler policy. It also respects Allow and Crawl-Delay directives. However, AWS notes that robots.txt parsing may not fully support all extended syntax (e.g., wildcards in paths are partially supported).

🔍 Detection Indicators

The primary User-Agent string is: Mozilla/5.0 (compatible; bedrockbot/1.0; +https://docs.aws.amazon.com/bedrock/latest/userguide/crawlers.html). Additional identifying headers include User-Agent: bedrockbot/1.0 and From: [email protected]. Behavioral fingerprints include consistent request timing (1 request/sec), use of AWS IP ranges, and a specific TLS fingerprint (TLS 1.3 with AWS cipher suites).

📊 Data Usage

Collected data is used exclusively for training and improving foundation models deployed on Amazon Bedrock, including models like Amazon Titan Text, Claude, and Llama. AWS states that data is anonymized and aggregated, and no personal identifiable information (PII) is intentionally collected. The data is not used for advertising or third-party resale.

⚙️ Rate Limiting Policy

Rate limiting is recommended because bedrockbot can generate sustained traffic during large-scale crawls, potentially impacting server performance. The policy rationale is to impose a threshold of 100 requests per minute per IP or 500 requests per hour across a subnet, above which blocking is triggered to protect origin infrastructure from excessive load.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.