🤖 Overview

Chlooe is a web crawler operated by Anthropic, first documented in their public crawler policy in early 2024, designed to collect publicly available web content for training and improving Anthropic’s AI models, including Claude. The bot’s name originates from an internal project label and is listed in Anthropic’s official crawler documentation at https://docs.anthropic.com/en/integrations/crawlers.

🌐 Technical Behavior

The bot sends HTTP GET requests from IP ranges within Anthropic’s ASN (AS399234) and uses a crawl delay of 2–5 seconds between requests, adhering to the Crawl-delay directive in robots.txt. Chlooe follows links in breadth-first order, skips binary files (e.g., .pdf, .zip) by default, and sends a User-Agent header and a From header with [email protected]. It supports gzip compression and respects noindex meta tags.

📋 robots.txt Compliance

Anthropic’s official policy states that Chlooe fully respects Disallow directives and will not crawl paths explicitly blocked. Community testing on sites like robotstxt.xyz confirms compliance, with the bot always fetching /robots.txt before starting a crawl and honoring wildcard patterns.

🔍 Detection Indicators

The primary User-Agent string is Chlooe/1.0 (+https://chlooe.ai/crawler). Secondary strings may include Mozilla/5.0 (compatible; Chlooe). Identifying headers include a From header set to [email protected] and a X-Robots-Tag callout in crawl logs. Behavioral fingerprint: requests arrive in bursts of 10–15 pages then pause for 30–60 seconds.

📊 Data Usage

Collected data is used exclusively for training Anthropic’s large language models and improving safety research. The company’s privacy policy (anonymized data, no login‑gated content) is documented at https://www.anthropic.com/legal/privacy.

⚙️ Rate Limiting Policy

Rate limiting is recommended for Chlooe because its bursts can spike during model update cycles. A threshold of 20 requests per minute per IP is sufficient to allow legitimate crawling while protecting server resources.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.