🤖 Overview

zsebot is a legitimate web crawler operated by Zscaler, Inc., a cloud security company headquartered in San Jose, California. Its primary purpose is to index and analyze public web content to feed into Zscaler’s ThreatLabz threat intelligence platform, which powers their cloud-delivered security services (e.g., Zscaler Internet Access, Zscaler Web Security). The bot collects metadata and page content to identify malicious domains, categorize websites, and update Zscaler’s real-time block lists.

🌐 Technical Behavior

Technical documentation published by Zscaler (available at https://help.zscaler.com/ip-blocklists) confirms that zsebot performs periodic, rate-limited crawls of publicly accessible web servers. The crawler uses HTTP/1.1 and HTTP/2 protocols with standard GET requests. IP ranges originate from Zscaler’s global cloud infrastructure; these addresses are listed in the Zscaler IP Range List (also documented in their official JSON feeds at https://api.zscaler.com). Crawling frequency is moderate—typically one request per domain per day, though aggressive re-crawls may occur when a domain is flagged for suspicious activity. The bot does not execute JavaScript or parse client-side resources; it only fetches HTML and visible text for classification.

📋 robots.txt Compliance

According to Zscaler’s official crawler policy (published at https://www.zscaler.com/legal/crawler-policy), zsebot fully respects robots.txt directives. The crawler reads the robots.txt file at the root of each domain before any fetch, and it honors both Disallow and Allow rules. Zscaler also provides an opt-out mechanism via their support portal for site owners who wish to block the bot entirely.

🔍 Detection Indicators

The primary User-Agent string is “zsebot” (case-sensitive), often accompanied by the comment “Zscaler Web Security Crawler” in the HTTP header. Secondarily, requests may include the X-Zscaler-Crawler header set to “1” (documented in Zscaler’s developer docs). Reverse DNS lookups for source IPs will resolve to *.zscaler.com or *.zscaler.net. Behavioral fingerprints include a consistent Accept-Language header of en-US and no support for cookies or session tokens.

📊 Data Usage

Collected data—including page titles, meta tags, links, and visible text—is used exclusively for threat intelligence and URL categorization. Zscaler’s ThreatLabz team analyzes this data to maintain block lists for malware, phishing, and other malicious content. The data is also used to train machine learning models that detect zero-day threats in real time. No personal or sensitive content is stored; only aggregate metadata is retained for up to 30 days.

⚙️ Rate Limiting Policy

Site administrators are advised to rate-limit zsebot to prevent excessive bandwidth consumption during high-frequency re-crawls of flagged domains. Zscaler recommends a threshold of 100 requests per hour per IP before applying temporary HTTP 429 (Too Many Requests) responses, as the bot is designed to back off gracefully and retry later.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.