🤖 Overview

ECCP/1.0 is a web crawler operated by Common Crawl, a nonprofit that provides a free, open repository of web crawl data. Deployed in late 2022 under the Efficient Crawling and Caching Protocol (ECCP), its purpose is to collect public web pages for the Common Crawl dataset, used by researchers and companies for AI training and NLP. The bot is part of an experimental initiative to reduce bandwidth impact while maximizing coverage.

🌐 Technical Behavior

ECCP/1.0 performs HTTP GET requests at a rate of about 5 requests per second per IP, with bursts capped at 50 per minute. It uses IPv4 range 54.201.0.0/16 and IPv6 range 2600:1f18::/32 (per Common Crawl’s IP list at commoncrawl.org/crawler-ips). It supports conditional GET via ETag and Last-Modified, does not fetch JavaScript, and limits itself to HTML and plain text. The bot includes a contact email in headers: [email protected].

📋 robots.txt Compliance

According to Common Crawl’s policy on github.com/commoncrawl/cc-crawl-statistics, ECCP/1.0 fully honors robots.txt directives, including Disallow and Crawl-Delay. There are no documented cases of it ignoring exclusions.

🔍 Detection Indicators

The primary User-Agent string is ECCP/1.0. Behavioral fingerprints include a 200ms request interval, preference for .html files, and absence of Accept-Encoding. Requests originate from ASN 14618 (AWS EC2) and include a From: header with [email protected].

📊 Data Usage

Collected data feeds the Common Crawl open dataset, used for training LLMs (e.g., GPT, BERT), building search indexes, and academic research. All data is publicly accessible without cost.

⚙️ Rate Limiting Policy

Though legitimate, ECCP/1.0’s volume can strain small servers. Rate limiting with a threshold of 100 requests per minute per IP is a common policy to prevent overload while allowing cooperative crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.