🤖 Overview

Scrubby is a web crawler operated by the CommonCrawl foundation, a non‑profit organization based in Redwood City, California, that maintains a free, open‑source repository of web crawl data. Scrubby is the primary crawler used to build the CommonCrawl dataset, which is released publicly every few months for research, AI training, and large‑scale web analysis. The bot’s official documentation is hosted on the CommonCrawl GitHub repository and its operation is described in the CommonCrawl technical papers.

🌐 Technical Behavior

Scrubby’s crawl pattern is based on a breadth‑first traversal of hyperlinks, starting from an initial seed list derived from the Alexa Top 1M and regular re‑crawls of previously discovered URLs. It issues requests at a moderate rate—typically 1–2 requests per second per IP—and uses persistent connections with HTTP/1.1. The IP ranges used by Scrubby are primarily announced by Amazon Web Services (AWS) and can be identified via the CommonCrawl IP list published at https://commoncrawl.org/ip‑ranges. Scrubby respects the robots.txt Crawl‑delay directive and will wait the specified number of seconds before the next request to the same host. It also supports the If‑Modified‑Since header to avoid re‑fetching unchanged content, reducing server load.

📋 robots.txt Compliance

According to CommonCrawl’s official documentation, Scrubby fully honors robots.txt Disallow directives and will not crawl any URL path blocked by the website administrator. The bot checks the robots.txt file at the start of each crawl session and caches it for the duration of the run. There are no documented cases of Scrubby intentionally ignoring robots.txt rules, and the foundation explicitly advises users to use standard robots.txt exclusions to control crawling.

🔍 Detection Indicators

Scrubby uses the User‑Agent string “Mozilla/5.0 (compatible; Scrubby/2.0; +http://commoncrawl.org/faq/)” (version numbers may vary). It also sends a From header with the value “[email protected]”. Behavioral fingerprints include a consistent crawl delay proportional to the Crawl‑delay directive, and sequential URL fetching without randomisation. The bot does not send cookies or accept‑language headers, simplifying identification.

📊 Data Usage

Data collected by Scrubby is used exclusively for the CommonCrawl public dataset, which is released under a Creative Commons license. The dataset contains raw HTML, metadata, and extracted text from billions of web pages, and is widely employed for training large language models (e.g., GPT‑3, BLOOM), academic research in information retrieval, and building web‑scale language corpora.

⚙️ Rate Limiting Policy

Although Scrubby is legitimate, its high frequency (multiple concurrent crawlers) and large crawl volume (terabytes per month) can stress under‑provisioned servers. Rate‑limiting is recommended to prevent excessive resource consumption while still allowing the bot to collect public data for its non‑profit mission. A threshold of 5–10 requests per minute per IP is typical, with a grace period for robots.txt compliance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.