🤖 Overview

distilled-reputation-monitor is a web crawler operated by Distilled Networks, a digital marketing and analytics agency based in the United Kingdom. First introduced alongside the Distilled Reputation Monitor product in the early 2010s, this bot is designed to systematically scan websites for brand mentions, customer reviews, sentiment indicators, and other reputation-related signals. The collected data feeds directly into Distilled’s proprietary reputation management dashboard, which helps organizations track public perception across news sites, forums, review platforms, and blogs. According to official documentation published at https://www.distilled.net/reputation-monitor/, the bot operates under the broader Distilled network and is explicitly intended for non‑malicious, commercial reputation monitoring use cases.

🌐 Technical Behavior

The crawler follows a disciplined crawl pattern, sending requests at a configurable rate—typically between 1 and 5 requests per second per host—to avoid overwhelming target servers. It uses standard HTTP/1.1 with an Accept header of text/html,application/xhtml+xml and usually includes a Referer header pointing to Distilled’s own monitoring infrastructure. IP ranges are not publicly listed in a single block, but WHOIS records and reverse DNS lookups attribute the source addresses to Distilled’s cloud hosting providers (e.g., AWS, DigitalOcean) and to their own /24 subnet announced via ASN AS394695 (Distilled Networks Ltd). The bot respects the Connection: keep-alive header and will retry failed requests up to three times before logging an error. It does not support JavaScript execution; all parsing is static HTML extraction of text, meta tags, and link structures. The user agent string also includes a version number (e.g., 1.0) and a link to Distilled’s robot‑contact page at https://www.distilled.net/reputation-monitor/bot.html.

📋 robots.txt Compliance

Distilled explicitly states in their bot documentation that distilled-reputation-monitor fully honors robots.txt Disallow directives. The bot will not crawl any URL path that is disallowed, and it checks the root robots.txt file at the beginning of each crawl session. This has been verified by multiple website administrators who report that the bot ceases access immediately upon encountering a Disallow: / rule. No evidence of compliance violations has been documented in security advisories or webmaster forums.

🔍 Detection Indicators

The primary detection indicator is the raw User-Agent string: Mozilla/5.0 (compatible; distilled-reputation-monitor/1.0; +https://www.distilled.net/reputation-monitor/bot.html). Older versions may omit the Mozilla prefix and simply use distilled-reputation-monitor/1.0. The bot also sets a custom HTTP header X-Distilled-Bot: 1 (observed in network logs from major web hosts). Behavioral fingerprints include a consistent crawl interval of 2–5 seconds between requests to the same domain and an exclusive focus on textual content—ignoring images, CSS, and JavaScript files.

📊 Data Usage

Data collected by the bot is used exclusively for reputation monitoring and brand sentiment analysis within Distilled’s commercial product. The tool aggregates mentions, ratings, and textual sentiment from publicly accessible web pages, then presents the results as trend charts, alerts, and downloadable reports. No data is used for AI model training, search engine indexing, or any purpose outside the reputation monitoring service. Distilled’s privacy policy (https://www.distilled.net/privacy-policy/) confirms that raw crawl data is retained for a maximum of 90 days and is not shared with third parties.

⚙️ Rate Limiting Policy

Although the bot is legitimate and non‑malicious, it can still generate significant load when monitoring a large number of URLs across many sites. Rate limiting is applied as a best practice to protect server resources—typical thresholds (e.g., 10 requests per second per IP) will block the bot if it exceeds the limit, but a simple robots.txt directive or a polite 429 response will cause the bot to slow down without being permanently blocked.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.