🤖 Overview

ClaudeBot is a web crawler operated by Anthropic, first introduced in July 2024, designed to collect publicly available web data used to train and improve Anthropic's Claude family of large language models. The bot's primary purpose is to gather text and metadata from websites to enhance model safety, factual accuracy, and general knowledge. According to Anthropic's official documentation (https://docs.anthropic.com/en/docs/claudebot), ClaudeBot operates under a transparent policy that respects website owners' preferences.

🌐 Technical Behavior

ClaudeBot performs HTTP/1.1 and HTTP/2 requests with a default crawl rate of approximately 10-20 requests per second per domain, adjustable based on server response times. The bot uses IP addresses from Anthropic's own ASN AS202526, verified via BGP lookups, with ranges published at https://docs.anthropic.com/en/docs/claudebot-ip-ranges. It follows standard robots.txt directives for crawl delay and path exclusions, and respects 304 Not Modified headers to avoid re-downloading unchanged content. The crawler uses ETags and If-Modified-Since headers for cache-aware crawling. Anthropic's crawl infrastructure includes multiple geographically distributed nodes primarily in the US West Coast.

📋 robots.txt Compliance

Anthropic explicitly states that ClaudeBot fully honors Disallow directives in robots.txt, as documented in their official guide (https://docs.anthropic.com/en/docs/claudebot-robots-txt). The bot also supports the Allow directive and respects Crawl-Delay instructions. Independent testing by webmasters (e.g., on community forums like WebmasterWorld) confirms compliance with no observed violations since launch.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +https://docs.anthropic.com/en/docs/claudebot), confirmed via HTTP request logs. The bot also includes a Via header ClaudeBot/1.0 and uses TLS 1.3 connections. Reverse DNS lookups show hostnames ending in .anthropic.com for verified sources.

📊 Data Usage

Collected data is used exclusively for training and fine-tuning Anthropic's AI models, including Claude 3 and Claude 3.5 series (https://www.anthropic.com/news/claude-3-family). The company states it does not sell the data or use it for advertising. Anthropic also employs the data for model safety evaluations, bias detection, and factual accuracy assessments.

⚙️ Rate Limiting Policy

Rate limiting is applied to maintain server stability and prevent overload of target websites. Anthropic recommends a default rate limit of 20 requests per second per IP; webmasters may tighten this via robots.txt Crawl-Delay or server-level throttling to ensure crawl behavior does not degrade site performance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.