🤖 Overview

Heritrix is an open-source, extensible web crawler developed and maintained by the Internet Archive, first released in 2004. It is the primary crawling engine used to collect snapshots of public web pages for the Wayback Machine and other digital preservation projects, enabling long-term archiving of web content for historical, research, and cultural heritage purposes.

🌐 Technical Behavior

Heritrix uses a highly configurable, Java-based architecture that supports both breadth-first and depth-first crawl strategies, with customizable scope policies. It typically sends requests with a configurable delay between fetches — the default range is 1 to 3 seconds, but operators can adjust based on the target site’s load tolerance. IP ranges used by official Internet Archive crawlers are published in the list of netblocks (e.g., 207.241.224.0/20, 208.70.24.0/21) as documented on the Archive’s robots.txt page and WHOIS records. Crawling favors HTTP/1.1 with keep-alive, and supports gzip compression. The crawler is designed to be polite by default, respecting robots.txt directives and Crawl-Delay headers, and will automatically throttle if it encounters frequent non‑200 responses or connection resets.

📋 robots.txt Compliance

Heritrix fully honors robots.txt directives by default. The Internet Archive explicitly documents that its crawler — identified by the User-Agent ia_archiver — obeys Disallow rules and the Crawl-Delay meta‑directive (archived at https://archive.org/about/exclude.html). Site owners can exclude their pages from the Wayback Machine by placing the noarchive meta tag or by adding a Disallow for ia_archiver in their robots.txt.

🔍 Detection Indicators

The primary User-Agent string for an Internet Archive crawl is Mozilla/5.0 (compatible; ia_archiver; +http://archive.org/about/terms.html). Variants exist such as ia_archiver/1.0 and Mozilla/5.0 (compatible; IA2; +http://archive.org/about/terms.html) for different crawling processes. Behavioral fingerprints include a consistent request pattern with a From header containing the operator’s email ([email protected]), and the absence of Accept-Encoding for many early‑generation crawls, though modern versions support gzip.

📊 Data Usage

Collected data is stored as WARC (Web ARChive) files and integrated into the Internet Archive’s Wayback Machine, providing publicly accessible historical snapshots of web pages. The data is also used for digital preservation, academic research on web evolution, and as part of the Archive‑It subscription service for institutions. No data is used for AI training or commercial analytics — the sole purpose is long‑term archival access at https://web.archive.org.

⚙️ Rate Limiting Policy

Heritrix is rate‑limited by default because it can produce high request volumes when crawling large, high‑value sites. Administrators typically enforce a per‑IP threshold (e.g., 10 requests per second) to protect server resources while still allowing polite archival crawls, since the crawler is designed to back off when it detects 429 or 503 responses. The policy rationale is to balance archival completeness with server load, ensuring the Wayback Machine can capture content without disrupting normal site operations.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.