🤖 Overview

Egothor is an open-source full-text search engine written in Java, originally created by Leo Galambos and maintained on GitHub (https://github.com/egothor/egothor). The project includes a dedicated web crawler component, the Egothor Crawler, designed to index web pages for use with the Egothor search engine library. It is a legitimate, rate-limited bot operated by individual site administrators or developers who deploy Egothor for internal or public site search functionality. The project documentation notes the crawler is not a commercial service but a free tool for building custom search indices.

🌐 Technical Behavior

The Egothor Crawler uses HTTP/1.1 GET requests and supports both HTTP and HTTPS protocols. By default, the crawler respects a configurable crawl delay (commonly set to 10 seconds between requests) as defined in its configuration file. It does not use a fixed set of IP ranges; instead, it operates from the IP address of the server on which it is deployed. The crawler can be configured to obey Crawl-Delay directives in robots.txt and supports URL filters to limit scope. Its request frequency is entirely user-defined, meaning it can be aggressive if misconfigured, but standard deployments throttle requests to avoid overloading servers. The crawler processes HTML pages, extracts links, text, and metadata, and can handle common content types like HTML and PDF. It does not execute JavaScript or parse dynamic content by default.

📋 robots.txt Compliance

The Egothor Crawler is documented as fully complying with the Robots Exclusion Standard (robots.txt). Its source code (available on GitHub) includes a parser that reads Disallow and Crawl-Delay directives and respects them unless the user overrides this behavior. The official documentation recommends always testing robots.txt conformance, and the project's issue tracker confirms it does not intentionally ignore directives. However, because the crawler is user-configured, improper setup could lead to non-compliance, but the default behavior honors all standard rules.

🔍 Detection Indicators

The primary User-Agent string used by the Egothor Crawler is EgothorCrawler/1.0 (Java; ...), with version numbers varying. It may also include the word "Egothor" in the User-Agent or From header. Behavioral fingerprints include sequential requests with consistent intervals, no referer header, and a low request rate in typical deployments. The crawler does not spoof its identity and is easily identifiable by the distinctive User-Agent pattern. Log entries showing EgothorCrawler are the primary detection method.

📊 Data Usage

The data collected by the Egothor Crawler is used exclusively for building an inverted index of web pages for use with the Egothor search engine. This index powers site search functionality on the operator’s own domain or within a closed network. There is no central server, no data sharing with third parties, and no use for AI training or commercial analytics. The project license (GNU General Public License) ensures the software remains open and free.

⚙️ Rate Limiting Policy

Rate limiting the Egothor Crawler is necessary because its crawl speed is entirely configurable by the operator; if a deployment sets a very low crawl delay (e.g., less than one second), it can overwhelm server resources. Threshold-based blocking protects the web application's availability while still allowing the legitimate indexing that the bot was designed to perform.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.