🤖 Overview

The Extractor is a web crawler operated by the Common Crawl Foundation, a nonprofit organization that maintains a free, open repository of web crawl data. Its primary purpose is to extract structured content (HTML, metadata, and text) from publicly accessible web pages to feed the Common Crawl dataset, which is used by researchers, AI developers, and data scientists for training large language models and conducting web-scale analysis.

🌐 Technical Behavior

The Extractor operates as part of Common Crawl’s pipeline (codenamed CCBot-2.0) and uses a custom HTTP client based on the Aspell library. It sends requests at a rate of approximately 1–2 requests per second per IP, with bursts of up to 5 requests in short intervals. The crawler uses a distributed architecture with IP ranges drawn from Amazon Web Services (AWS) and Google Cloud Platform (GCP) — specifically the 52.x.x.x and 35.x.x.x blocks. It supports both HTTP/1.1 and HTTPS, and it requests pages with a default Accept-Language: en-US,en;q=0.9 header. The Extractor does not follow meta refresh redirects and ignores rel="nofollow" links, but it does respect the nofollow meta tag value in robots meta tags.

📋 robots.txt Compliance

According to official Common Crawl documentation (available at https://commoncrawl.org/robots.txt), the Extractor fully obeys the robots.txt exclusion protocol. It caches robots.txt for up to 24 hours and re-fetches it after expiration. However, it does not obey X-Robots-Tag HTTP headers unless the tag is explicitly supported (e.g., noindex is ignored). The crawler also respects the Crawl-Delay directive when present, but only with a minimum delay of 10 seconds.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; CCBot/2.0; +https://commoncrawl.org/faq/) — note the variation "CCBot" instead of "Extractor" in version 2.0. Additional fingerprints include a distinctive User-Agent: Extractor/1.0 for older deployments (pre‑2020) and a From: [email protected] header. The crawler does not set a Referer header and always sends a Connection: close request header.

📊 Data Usage

All content retrieved by Extractor becomes part of the Common Crawl dataset, which is freely downloadable from their official S3 bucket (s3://commoncrawl/). The extracted data — including raw HTML, extracted text, and metadata (e.g., page title, timestamp, and outgoing links) — is used for AI training, natural language processing research, search engine indexing by third parties, and web analytics as described in their 2023 usage policy (https://commoncrawl.org/use).

⚙️ Rate Limiting Policy

The Extractor is rate‑limited because its aggressive crawl schedule (up to 10,000 pages per hour per instance) can overwhelm small servers. IT administrators are advised to implement threshold‑based blocking (e.g., trigger a 429 response after 50 requests per minute from a single IP) to protect site performance without permanently banning the bot, which remains a legitimate and valuable data source.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.