🤖 Overview

t-h-u-n-d-e-r-s-t-o-n-e is the designated User‑Agent for the web crawler operated by Thunderstone Software LLC, a company headquartered in Cleveland, Ohio, that develops enterprise search and indexing solutions. This bot is the primary agent used to collect publicly accessible web content for Thunderstone’s search products, including the Webinator and Texis platforms, which power custom search engines for corporate intranets and public websites. According to Thunderstone’s official documentation, the crawler is designed to be respectful of server resources while enabling comprehensive indexing for client deployments.

🌐 Technical Behavior

The crawler follows standard HTTP/1.1 protocols and supports both GET and HEAD requests, with a configurable crawl delay typically set to 2–5 seconds between requests to the same host. Thunderstone provides a list of originating IP ranges in their support portal (e.g., 208.73.210.0/24 and 64.94.46.0/24) which may change over time. The bot identifies itself via the User‑Agent string "Mozilla/5.0 (compatible; Thunderstone/10.0; +http://www.thunderstone.com/)" and sends a From header containing the operator’s contact email. It also respects the Content‑Type header and only indexes HTML, XML, and plain‑text content by default, though it can be configured to parse PDF and Office documents via Texis extensions.

📋 robots.txt Compliance

Thunderstone explicitly states that their crawler fully honors the Robots Exclusion Protocol (robots.txt) and supports the Crawl‑Delay directive. Official documentation from Thunderstone’s website (www.thunderstone.com) advises webmasters to use User‑agent: Thunderstone or User‑agent: t‑h‑u‑n‑d‑e‑r‑s‑t‑o‑n‑e in their robots.txt file to apply rules. There have been no reported violations of Disallow directives in security advisories or community forums.

🔍 Detection Indicators

The primary identification string is Thunderstone/10.0 (version may vary; e.g., 8.0, 9.0) and the exact agent t‑h‑u‑n‑d‑e‑r‑s‑t‑o‑n‑e used in the User‑Agent header. Behavioral fingerprints include low request frequency, a consistent Accept‑Encoding: gzip header, and the absence of Referer or Cookie headers. The bot also sends a User‑Agent: Mozilla/5.0 (compatible; Thunderstone) prefix to mimic a common browser, making header‑based filtering essential.

📊 Data Usage

Data collected by the Thunderstone crawler is used exclusively for search indexing within Thunderstone’s enterprise and public search products. The indexed content is stored locally on client servers—not sent to Thunderstone’s cloud—and is never used for AI training or behavioral analytics. Thunderstone’s privacy policy confirms that no personal data is permanently retained beyond the index refresh cycle.

⚙️ Rate Limiting Policy

While this bot is legitimate and rate‑limited by design (default crawl delay of 2–5 seconds), administrators may implement threshold‑based blocking if the bot fails to respect custom Crawl‑Delay values set in robots.txt. Such blocking is a preventive measure to protect server stability from misconfigured or aggressive instances, not a reflection of malicious intent.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.