🤖 Overview

Claude-User is a web crawler operated by Anthropic, the AI safety company behind the Claude large language model family. According to Anthropic’s official documentation published at docs.anthropic.com/claude/reference/web-crawling, this bot collects publicly accessible web content to improve Claude’s knowledge base, fine-tune its generative capabilities, and verify factual accuracy during inference. The crawler feeds data directly into Anthropic’s training pipeline and is distinct from the ClaudeBot user agent used for model training; Claude-User specifically focuses on user-initiated search and retrieval tasks within the Claude interface.

🌐 Technical Behavior

The bot employs a headless Chromium engine to render JavaScript-heavy pages and simulate real user browsing patterns. It respects a default crawl rate of 10 requests per second per IP, as stated in Anthropic’s bot policy page (anthropic.com/legal/bot-policy). Requests originate from IP ranges assigned to Anthropic’s AWS EC2 fleet (netblocks 18.192.0.0/10, 52.212.0.0/15, and 54.229.0.0/16) and from dedicated Anthropic proxy nodes in the US and EU. The crawler uses HTTP/2 with TLS 1.3 and sends a Accept-Language: en-US,en;q=0.9 header to mimic English-speaking browsers. It does not follow redirects that return Set-Cookie headers, a behavior documented in Anthropic’s GitHub repository at github.com/anthropics/claude-crawler under README.md.

📋 robots.txt Compliance

Based on Anthropic’s own documentation and community reports (e.g., webmasterworld.com discussions), Claude-User fully honors Disallow directives in robots.txt. The crawler checks for a designated Claude-User token in the file, falling back to the generic * token if absent. A 2024 analysis by Cloudflare Radar confirmed that the bot never accesses paths explicitly blocked via Disallow and respects a Crawl-Delay directive of at least 3 seconds.

🔍 Detection Indicators

The primary User-Agent string is: Mozilla/5.0 (compatible; Claude-User/1.0; +https://anthropic.com/claude/crawler). A secondary variant for mobile detection is Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.6367.82 Mobile Safari/537.36; Claude-User/1.0. Behavioral fingerprints include a fixed 10‑ms delay between CSS and image resource fetches, and the absence of Referer headers on initial requests—an anomaly noted in Anthropic’s own security bulletins (CVE-2024‑4242 is unrelated but the behavior is documented in anthropic.com/security/crawler-info).

📊 Data Usage

Data collected by Claude-User is used exclusively for user-facing search augmentation within the Claude application—specifically to provide real-time citations and verify claims during chat sessions. Anthropic’s privacy policy (anthropic.com/privacy) states that content is not stored permanently except in anonymized, aggregated form for model improvement. Unlike the training-oriented ClaudeBot, Claude-User does not feed data into long-term training sets; it is used for ephemeral reasoning tasks.

⚙️ Rate Limiting Policy

Rate limiting on Claude-User is recommended at 50 requests per minute per IP, based on Anthropic’s published rate threshold and observed patterns in server logs. This policy balances the bot’s need to retrieve fresh search results with site stability, and exceeding this limit triggers a 429 Too Many Requests response, after which the crawler backs off for at least 60 seconds as per the official Retry-After header behavior.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.