🤖 Overview

The Charlotte web crawler, identified by User-Agent CCBot, is operated by the Common Crawl Foundation, a nonprofit based in the U.S. since 2011. Its purpose is to produce a free, open archive of the public web, which is widely used for training AI models (e.g., GPT, LLaMA), academic research, and search indexing.

🌐 Technical Behavior

Charlotte is built on the Apache Nutch framework with a custom scheduler. It performs breadth-first crawls from a seed list of high-authority domains, following internal links up to depth 10. It uses HTTP/1.1 with conditional GETs (ETag, Last-Modified) to minimize bandwidth. Request rate is ~1–2 per second per domain. IP ranges are published at commoncrawl.org/ip-ranges, including 52.70.242.0/24 and 54.94.196.0/24 (AWS us-east-1). It does not execute JavaScript or set cookies.

📋 robots.txt Compliance

According to Common Crawl’s FAQ, CCBot fully respects robots.txt as per RFC 9309. It follows Disallow, Allow, and Crawl-Delay directives. The crawler also supports the X-Robots-Tag HTTP header for per-page control. Compliance is verified by third-party audits.

🔍 Detection Indicators

The primary User-Agent string is CCBot/2.0 (https://commoncrawl.org/faq/). Older versions use CCBot/1.0. Behavioral fingerprints include steady request rates, high proportion of HTML pages, and origin from known AWS IP ranges. No POST methods are used.

📊 Data Usage

Collected data is released monthly as WARC files on Amazon S3, along with WAT metadata and WET text extracts. The dataset, exceeding 100 TB, is used for training language models, web graph analysis, and information retrieval research. Common Crawl also provides the CC-MAIN index for programmatic access.

⚙️ Rate Limiting Policy

Although legitimate, Charlotte’s crawl volume can be high during fresh runs. Security teams often apply rate limiting (e.g., 15–20 RPM from CCBot) to protect server resources. Common Crawl recommends using Crawl-Delay in robots.txt instead of IP blocks to maintain dataset representativeness.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.