🤖 Overview

Tarantula is a web crawler operated by the Common Crawl Foundation, a non‑profit organization that maintains a free, open repository of web crawls. First documented in 2012, Tarantula is one of the crawlers used by the Foundation to collect publicly accessible web pages for large‑scale research, AI training, and data analytics. It feeds data into the publicly available Common Crawl dataset, which is used by academic institutions, AI labs, and companies for language model training, search indexing, and web science.

🌐 Technical Behavior

Tarantula crawls using a distributed architecture based on the open‑source Heritrix crawler engine, with custom modifications for scale and politeness. Request frequency is typically limited to one request per second per domain, though the crawler may burst to several requests per second during deep scans of large sites. IP ranges are drawn from the Common Crawl Foundation’s announced autonomous system (e.g., ASN 396982 for AWS‑hosted crawlers) and from a pool of Amazon Web Services and Google Cloud Platform addresses. Tarantula supports HTTP/1.1 and HTTP/2, sends a standard Accept header, and includes a From header with the email address [email protected] on some requests. It respects Cache‑Control and Etag headers to reduce load on origin servers.

📋 robots.txt Compliance

Tarantula honors robots.txt directives as specified in the Internet Engineering Task Force (IETF) standard (RFC 9309). According to the Common Crawl codebase on GitHub (github.com/commoncrawl/cc-crawler), the crawler reads the Disallow rules for each host before beginning a crawl session. Disallowed paths are skipped, and the crawler respects a Crawl‑Delay directive if present. However, the Foundation notes that some operators mistakenly block all robots, so they recommend using targeted Disallow for sensitive paths rather than a blanket deny.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; CCBot/2.0; +http://commoncrawl.org/faq/) for the main crawler, but Tarantula may also appear as Mozilla/5.0 (compatible; Tarantula/1.0; +http://commoncrawl.org) in legacy deployments. Additional identifying headers include Via: tarantula and User‑Agent: CommonCrawl. Behavioral fingerprints include a consistent crawl depth limit of three hops from seed URLs, a typical request interval of one second, and DNS lookups that resolve to AWS or GCP IPs.

📊 Data Usage

All data collected by Tarantula is published as part of the Common Crawl dataset, available for free download under a Creative Commons license. The dataset is used for training large language models (e.g., GPT‑3, BLOOM, and other open‑source models), search engine indexing, web graph analysis, and archival research. The Foundation does not sell the data directly but relies on donations and grants to fund its operations.

⚙️ Rate Limiting Policy

Although Tarantula is a legitimate, rate‑limited crawler, administrators may choose to rate‑limit it to prevent excessive bandwidth consumption during peak times or to protect fragile back‑end services. A threshold‑based block (e.g., allowing up to 5 requests per second per IP) is recommended because the crawler’s polite behavior already keeps load low, and aggressive blocking could degrade the quality of open web research.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.