🤖 Overview

htdig (HT://Dig) is an open-source web indexing and search system originally developed by Andrew Scherpbier in the mid-1990s and later maintained by the htdig Community. Its primary purpose is to crawl and index web content for site‑internal search engines—commonly deployed on university, corporate, or personal websites that require a self‑hosted, full‑text search solution without relying on external search providers. The software is written in C++, uses a backend database (typically Berkeley DB), and is distributed under a BSD‑style license. Official documentation and source code are archived at the SourceForge project page (https://sourceforge.net/projects/htdig/), where the latest stable release (3.2.0b6, dated 2004) remains available, though the project is no longer actively maintained.

🌐 Technical Behavior

htdig performs recursive HTTP/HTTPS crawls starting from a configurable seed URL list, respecting standard web protocols. It sequentially fetches pages, extracts hyperlinks, and follows them up to a configurable depth (default 2), while limiting the total number of pages (default 1000). The crawler uses a single‑threaded, non‑concurrent approach, making sequential requests without parallelization—unlike modern bots. Default request frequency is not throttled, but administrators can configure a delay (–c parameter) between requests. IP ranges are not fixed; the crawler uses the IP address of the machine running it, typically within private ranges for intranet use. It supports HTTP/1.0 and HTTP/1.1 with basic authentication and cookie handling via configuration files. The crawler indexes MIME types (text/html, text/plain, application/pdf via external converter) and parses HTML to build an inverted index. No JavaScript rendering is performed. According to the official documentation, htdig does not support HTTPS natively in older versions, but can be compiled with OpenSSL.

📋 robots.txt Compliance

htdig fully honors robots.txt directives as specified by the Robots Exclusion Protocol. The crawler reads the /robots.txt file from each host it visits before fetching any resource, and it respects both Disallow and Allow rules. This behavior is documented in the htdig reference manual (available on SourceForge) under the “robots.txt” configuration section. If the server returns a 404 for robots.txt, the crawler proceeds with no restrictions. The default User‑Agent string used for matching robots.txt is “htdig/3.2.0b6” (or version‑specific).

🔍 Detection Indicators

The primary identification is the User‑Agent header: typical strings are htdig/3.2.0b6 or htdig/3.2.0. Older releases may use htdig/3.1.x. The crawler does not set custom X‑headers or specific connection patterns. Behavioral fingerprints include sequential, non‑concurrent requests from a single IP, with no Referer or Accept‑Language headers (unless configured). The default request uses GET and reads minimal HTTP metadata. A definitive test is to check for the absence of modern client features (e.g., no JavaScript, low request rate).

📊 Data Usage

Data collected by htdig is used exclusively for the operator’s own internal site search engine. The indexed content is stored locally in Berkeley DB files and served via a CGI script (htsearch) that provides a search interface. No data is transmitted to third parties; the system is fully self‑contained. Typical use cases include intranet portals, academic department websites, and archives where a lightweight, offline search solution is needed.

⚙️ Rate Limiting Policy

htdig is rate‑limited because its default configuration does not include a built‑in delay between requests, which can overwhelm under‑provisioned servers if left unrestricted. Administrators are advised to configure a crawl delay of at least 1–2 seconds per request via the –d or delay configuration option. Since the bot is not malicious but can be aggressive when misconfigured, threshold‑based blocking (e.g., >100 requests/min from one IP) is a reasonable policy to prevent resource exhaustion while still allowing legitimate indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.