🤖 Overview

my-heritrix-crawler is a custom user agent string typically associated with instances of the Heritrix web crawler, an open-source, extensible crawler developed and maintained by the Internet Archive. First released in 2004 and hosted at https://github.com/internetarchive/heritrix3, Heritrix is designed for high-fidelity archival web harvesting, not for commercial search indexing or AI training. The “my-” prefix indicates a personalized deployment, often used by libraries, universities, or researchers conducting focused crawls for digital preservation projects. Unlike standard commercial bots, this agent is not operated by a single organization; its behavior depends entirely on the configuration of the individual Heritrix instance running it.

🌐 Technical Behavior

Heritrix crawls using a modular architecture with configurable crawl profiles, supporting HTTP/1.1, HTTPS, IPv4, and IPv6. It respects robots.txt by default but can be overridden for authorized crawls. Request frequency is defined by user-set “politeness” delays, typically ranging from 1 to 30 seconds between requests per host, controlled by the CrawlController settings. IP ranges are not fixed; they correspond to the server(s) hosting the crawler, often within academic or institutional blocks. The crawler sends a robust set of HTTP headers, including User-Agent: my-heritrix-crawler, From (if configured), and Accept values typical of modern browsers. It can perform deep recursive crawling, downloading embedded resources like CSS, JavaScript, and images, and can be set to honor nofollow and noindex directives. The official documentation at the Internet Archive’s wiki (https://webarchive.jira.com/wiki/display/Heritrix) details its multi-threaded crawling engine, which can be tuned to be aggressive or gentle.

📋 robots.txt Compliance

By default, Heritrix respects robots.txt directives, including Disallow and Crawl-delay, as documented in its operator’s guide. However, because my-heritrix-crawler is a custom deployment, compliance is not guaranteed—site operators should check the From header or contact the administrator listed in the User-Agent string if present. The Internet Archive’s own crawlers follow strict robots.txt rules, but third-party instances may override them for internal research.

🔍 Detection Indicators

The primary identifier is the User-Agent string my-heritrix-crawler (often followed by version info, e.g., my-heritrix-crawler/3.4.0). Additional headers may include From: [email protected] if configured. Behavioral fingerprints include sequential requests with consistent delays, lack of JavaScript execution, and a Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 pattern typical of Heritrix.

📊 Data Usage

Data collected by my-heritrix-crawler is used exclusively for the operator’s own archival or research purposes—such as building web collections, digital preservation, or academic studies. It is not sold or fed into commercial AI models. The Internet Archive uses Heritrix for the Wayback Machine, but public instances are independent.

⚙️ Rate Limiting Policy

Because my-heritrix-crawler can be configured to crawl aggressively (e.g., with no delay), it is prudent to rate-limit it using threshold-based blocking (e.g., 50 requests per minute per IP) to protect server resources and maintain site stability for other users, while still allowing legitimate, polite crawls.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.