🤖 Overview

Cvaulev is a web crawler operated by Anthropic, the AI research company behind the Claude model family. It was first identified in early 2024 and is used to collect publicly available web content for training Anthropic's large language models, including Claude 3 and subsequent versions. The crawler is part of Anthropic's data collection pipeline, which emphasizes responsible AI development and transparency.

🌐 Technical Behavior

The bot typically sends requests from IP ranges within the Anthropic network, which are not publicly listed but are identifiable through reverse DNS lookups. It uses HTTP/1.1 and HTTP/2 protocols, with a default crawl rate of approximately 1 request per second per host, though this may vary based on server response times. The crawler follows a breadth-first strategy and respects robots.txt crawl-delay directives when present. It does not fetch binary content such as images or videos unless explicitly required for text extraction. The user agent string includes the token Claude-Web, and the bot identifies itself via a User-Agent header formatted as follows: Mozilla/5.0 (compatible; Claude-Web/1.0; +https://anthropic.com/robot). According to Anthropic's official documentation, the bot does not bypass rate limits or employ aggressive caching techniques.

📋 robots.txt Compliance

Anthropic states that Cvaulev honors robots.txt directives, including Disallow and Crawl-Delay rules, as verified in their public policy page at https://anthropic.com/robots.txt. The bot will not access any URL path that is disallowed, and it respects the User-agent line for Claude-Web. There is no evidence of non-compliance in security advisories or community reports.

🔍 Detection Indicators

The primary indicator is the User-Agent string Claude-Web/1.0 with a comment linking to Anthropic's robot policy page. Additional detection can be performed by analyzing the Accept header, which typically includes text/html,application/xhtml+xml. The bot's IP ranges are not publicly documented, but network administrators can identify traffic by the Anthropic ASN (if known) or by the presence of the Claude-Web token. Behavioral fingerprints include a consistent request interval and no support for gzip or deflate compression in some versions.

📊 Data Usage

Collected web content is used exclusively for training and improving Anthropic's AI models, including Claude 3 Opus, Sonnet, and Haiku. The data helps the models learn language patterns, factual knowledge, and reasoning abilities. Anthropic's privacy policy states that they do not use personal data from crawls to identify individuals, and they provide an opt-out mechanism via robots.txt.

⚙️ Rate Limiting Policy

Rate limiting is recommended for this bot because its steady crawl rate can still accumulate significant load over time on high-traffic sites. A threshold-based blocking policy that returns 429 Too Many Requests after a certain number of requests per minute is appropriate, as it preserves site performance while still allowing legitimate data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.