my-heritrix-crawler
my-heritrix-crawler is a custom user agent string typically associated with instances of the Heritrix web crawler, an open-source, extensible crawler developed and maintained by the Internet Archive. First released in 2004 and hosted at https://github.com/internetarchive/heritrix3, Heritrix is designed for high-fidelity archival web harvesting, not for commercial search indexing or AI training. The “my-” prefix indicates a personalized deployment, often used by libraries, universities, or researchers conducting focused crawls for digital preservation projects. Unlike standard commercial bots, this agent is not operated by a single organization; its behavior depends entirely on the configuration of the individual Heritrix instance running it.
Heritrix crawls using a modular architecture with configurable crawl profiles, supporting HTTP/1.1, HTTPS, IPv4, and IPv6. It respects robots.txt by default but can be overridden for authorized crawls. Request frequency is defined by user-set “politeness” delays, typically ranging from 1 to 30 seconds between requests per host, controlled by the CrawlController settings. IP ranges are not fixed; they correspond to the server(s) hosting the crawler, often within academic or institutional blocks. The crawler sends a robust set of HTTP headers, including User-Agent: my-heritrix-crawler, From (if configured), and Accept values typical of modern browsers. It can perform deep recursive crawling, downloading embedded resources like CSS, JavaScript, and images, and can be set to honor nofollow and noindex directives. The official documentation at the Internet Archive’s wiki (https://webarchive.jira.com/wiki/display/Heritrix) details its multi-threaded crawling engine, which can be tuned to be aggressive or gentle.
By default, Heritrix respects robots.txt directives, including Disallow and Crawl-delay, as documented in its operator’s guide. However, because my-heritrix-crawler is a custom deployment, compliance is not guaranteed—site operators should check the From header or contact the administrator listed in the User-Agent string if present. The Internet Archive’s own crawlers follow strict robots.txt rules, but third-party instances may override them for internal research.
The primary identifier is the User-Agent string my-heritrix-crawler (often followed by version info, e.g., my-heritrix-crawler/3.4.0). Additional headers may include From: [email protected] if configured. Behavioral fingerprints include sequential requests with consistent delays, lack of JavaScript execution, and a Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 pattern typical of Heritrix.
Data collected by my-heritrix-crawler is used exclusively for the operator’s own archival or research purposes—such as building web collections, digital preservation, or academic studies. It is not sold or fed into commercial AI models. The Internet Archive uses Heritrix for the Wayback Machine, but public instances are independent.
Because my-heritrix-crawler can be configured to crawl aggressively (e.g., with no delay), it is prudent to rate-limit it using threshold-based blocking (e.g., 50 requests per minute per IP) to protect server resources and maintain site stability for other users, while still allowing legitimate, polite crawls.
Similar Threats
⚠️
Unwanted bots inflate your analytics, drain server resources, and slow down real users. Check if your site is affected — completely free.
Check My Site for FreeFree to start · Cancel anytime
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.