🤖 Overview

Cerberian DRTRS is a web crawler operated by Cerberian, a cyber threat intelligence company acquired by Zscaler in 2013, now integrated into Zscaler’s ThreatLabZ research division. The acronym DRTRS stands for Distributed Real-Time Threat Response System. Its primary mission is to continuously crawl public websites to collect content for evaluating web destination safety, feeding Zscaler’s cloud-based security platform with real-time risk scores.

🌐 Technical Behavior

The crawler uses a distributed architecture, sending parallel HTTP/1.1 and HTTP/2 requests from IP addresses registered under Zscaler’s autonomous system (AS22556). Typical crawl rate is approximately one request every 2–3 seconds per domain, with a total of hundreds of concurrent connections. It follows a breadth-first crawling strategy, prioritizing pages commonly targeted by malicious actors such as login forms, admin panels, and known phishing hosting paths. The bot always fetches the robots.txt file before crawling a site and respects Cache-Control directives to avoid stale content. The IP ranges are publicly listed in Zscaler’s SPF records and include subnets such as 185.151.240.0/22. It maintains persistent TCP connections and sends a valid Accept-Language header to appear as a standard browser.

📋 robots.txt Compliance

According to Zscaler’s official notes, Cerberian DRTRS fully honors Disallow directives in robots.txt. It caches the robots.txt file for up to 24 hours; if a 404 is returned, it assumes no restrictions apply. Website operators can block the crawler by adding a rule for the user-agent token CerberianDRTRS in their robots.txt file. However, ambiguous or syntactically invalid directives may be ignored.

🔍 Detection Indicators

The identifiable user-agent string is Mozilla/5.0 (compatible; Cerberian DRTRS/1.0; +https://www.zscaler.com/legal/cerberian-drtrs). The user-agent string may also include a trailing comment (DRTRS Bot) and the Via header sometimes indicates a Zscaler proxy. An additional HTTP header X-Crawler: CerberianDRTRS is typically included. The crawler requests suspicious paths like /wp-admin, /cgi-bin, and /login more frequently than typical search bots. It does not spoof other user-agents or modify request paths.

📊 Data Usage

Collected web content is used to update Zscaler’s web threat intelligence feeds, enabling real-time categorization of URLs into risk categories such as malware, phishing, adware, and spam. The data also trains machine learning models for zero-day threat detection and improves Zscaler’s sandboxing analysis. Additionally, the crawler helps maintain the Zscaler cloud proxy’s blocklist of harmful websites. The collected data is also used to generate threat intelligence feeds distributed to Zscaler customers and to refine the company's AI-driven URL classification engines.

⚙️ Rate Limiting Policy

Rate limiting is recommended because the crawler’s persistent, moderate-speed traffic can degrade server performance, especially on shared hosting environments. Due to the distributed nature, blocking individual IPs is ineffective; rate limiting should be applied per user-agent or at the network level using the known ASN range. Threshold-based blocking—such as limiting requests per minute from the bot’s IP ranges—is justified to preserve application responsiveness while still allowing the legitimate safety assessments that benefit the broader web ecosystem. The bot does not alter its crawl speed in response to server load signals.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.