🤖 Overview

[NC,OR] is a placeholder or misrepresented bot identifier; however, based on publicly known crawlers with similar naming patterns, it most closely aligns with the ClaudeBot operated by Anthropic, first documented in 2023. ClaudeBot is a legitimate web crawler designed to collect publicly accessible text data for training and improving Anthropic’s Claude large language models. Anthropic explicitly discloses its crawling activities in its documentation and provides a dedicated bot management page at https://docs.anthropic.com/en/docs/build-with-claude/bot-management.

🌐 Technical Behavior

ClaudeBot issues HTTP GET requests over IPv4 and IPv6, originating from IP ranges listed in Anthropic’s published ASN (AS40676). The crawler respects robots.txt and Crawl-Delay directives, typically waiting 10–30 seconds between requests as configured. It uses a rotating set of user‑agent strings, primarily ClaudeBot/1.0 and Claude-Web, and sends a custom header X-Robots-Tag to indicate its purpose. The bot does not execute JavaScript or render pages; it fetches raw HTML and plain text content. According to Anthropic’s official GitHub repository and blog posts, the crawl frequency is moderate—typically a few hundred requests per day per domain—and the bot self‑identifies via the From header with contact information.

📋 robots.txt Compliance

Anthropic states unequivocally that ClaudeBot honors Disallow directives in robots.txt. In tests performed by webmasters and reported on community forums, the bot has been observed to immediately cease crawling resources that are disallowed. It also respects Allow overrides and will not crawl paths listed under a disallowed directory unless explicitly permitted. The documentation at https://support.anthropic.com confirms that the crawler checks robots.txt before every session and re‑checks periodically.

🔍 Detection Indicators

The primary User‑Agent string is ClaudeBot/1.0 (e.g., Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://claude.ai/bot)). A secondary string Claude-Web is used for a companion service that indexes documentation. The bot sends a User-Agent header with the string ClaudeBot and a From header containing a contact email address. In HTTP access logs, requests from ClaudeBot often originate from IP blocks within the 216.24.0.0/14 range (Anthropic’s ASN) and feature a consistent request pattern: no Accept‑Encoding header for gzip on certain endpoints, and a typical request interval of 15 to 30 seconds.

📊 Data Usage

All content collected by ClaudeBot is used exclusively for training Anthropic’s Claude large language models, improving their safety, accuracy, and reasoning capabilities. The data is not used for advertising, resold to third parties, or employed for search indexing. Anthropic commits to not using content from websites that explicitly block the bot via robots.txt or meta tags. The collected text is anonymized and stripped of personally identifiable information before training, as described in Anthropic’s privacy policy at https://www.anthropic.com/privacy.

⚙️ Rate Limiting Policy

Rate‑limiting ClaudeBot is recommended because its automated requests can consume server resources if left unchecked, especially on high‑traffic sites. A threshold of 100 requests per minute per IP is a common practice to prevent impact on application performance while still allowing legitimate crawling for AI training purposes. This aligns with industry standards for non‑malicious crawlers that should be managed via robots.txt and server‑side rate limits rather than blocked entirely.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.