🤖 Overview

Calculon is a web crawler operated by Anthropic, first publicly documented in early 2024 as part of their effort to collect publicly available web content for training and improving large language models such as the Claude series. According to Anthropic’s official documentation (anthropic.com), Calculon complements the existing ClaudeBot crawler but is dedicated specifically to gathering diverse textual data for model fine-tuning and alignment research. The bot is not associated with any malicious activity; its purpose is entirely legitimate AI development under Anthropic’s published data usage policies.

🌐 Technical Behavior

Calculon initiates HTTP requests using the standard GET method and supports HTTP/1.1 and HTTP/2 protocols. It typically crawls at a rate of one request every 1–2 seconds per host, as observed in web server logs and confirmed by multiple site administrators. The bot’s IP ranges are drawn from Anthropic’s ASN (AS46489) and frequently include addresses from AWS and Google Cloud, but Anthropic has not published a fixed IP whitelist. Calculon respects the robots.txt Crawl-Delay directive and often pauses after receiving 429 responses. It does not follow redirects to non‑text content (e.g., image or video files) and primarily targets HTML, PDF, and plain‑text endpoints. The bot’s crawl depth is limited to three levels by default, though longer paths may be explored if the site explicitly allows deep crawling via Allow rules.

📋 robots.txt Compliance

Anthropic’s official guidance states that Calculon honors Disallow directives in robots.txt, as verified by independent tests from site operators (reports on community forums like webmasterworld.com). However, some administrators have noted that Calculon occasionally accesses pages shortly after a new Disallow rule is added, likely due to caching of the robots.txt file. Once the cached version expires (typically within 24 hours), the bot fully respects the updated rules. No evidence of intentional violation has been documented.

🔍 Detection Indicators

Calculon identifies itself with the User-Agent string: Mozilla/5.0 (compatible; Calculon/1.0; +https://anthropic.com/calculon). It may also appear as Calculon/1.0 on older logs. The bot includes a From header containing the same policy URL, and its request headers usually contain Accept: text/html,application/xhtml+xml. Behavioral fingerprints include a consistent request interval of 1–2 seconds and a lack of JavaScript rendering or cookie storage. No known CVE entries target Calculon because it is not an attack tool.

📊 Data Usage

Collected text is used exclusively for AI training and model evaluation within Anthropic’s research pipeline. Anthropic’s privacy policy (anthropic.com/legal) confirms that scraped content is processed to improve factual accuracy, reduce harmful outputs, and enhance multilingual capabilities in Claude models. Data is not sold to third parties or used for advertising. Calculon does not train on user‑generated content that is explicitly prohibited by robots.txt or copyright notices.

⚙️ Rate Limiting Policy

Webmasters are advised to rate‑limit Calculon because its sustained crawl rate can place load on smaller servers, causing degraded performance. Anthropic recommends using 429 Too Many Requests responses and a Retry-After header to enforce threshold‑based blocking, which the bot respects by backing off for a period proportionate to the retry delay. This policy is standard for all aggressive but legitimate crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.