🤖 Overview

WebZip is a web crawler developed by Spidersoft, a software company based in the United States that produces offline browsing and website mirroring tools. Originally released in the early 2000s, WebZip is designed to download entire websites—including HTML pages, images, CSS, JavaScript, and other assets—for local archival and offline viewing by individual users.

🌐 Technical Behavior

WebZip operates as a multi-threaded HTTP client that can open up to hundreds of simultaneous connections to a single server, often without built-in delay mechanisms. It sends standard GET requests and parses robots.txt on startup, but its default configuration does not enforce a crawl delay (Crawl-Delay directive). The crawler uses the HTTP/1.1 protocol and supports cookies, referrer headers, and conditional GETs (If-Modified-Since). IP addresses come from the user’s residential or business network, not from a fixed, published range—making geoblocking ineffective against it.

📋 robots.txt Compliance

According to Spidersoft’s official documentation (archived at spidersoft.com/webzip/help/), WebZip respects Disallow and Allow directives by default when the “Honor robots.txt” option is enabled in the application settings. However, users can disable this feature, causing the crawler to ignore robots.txt entirely. Site administrators should therefore assume that WebZip may not always honor exclusions.

🔍 Detection Indicators

The primary identifying User-Agent string is “WebZip/7.0” (or “WebZip/6.0”, “WebZip/8.0”), often appended with browser-like tokens such as “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; WebZip/7.0)”. Behavioral fingerprints include a rapid burst of requests to many different URLs in a short period, often targeting static assets like images and PDFs, and a lack of a common referrer chain.

📊 Data Usage

Collected website content is stored exclusively on the user’s local machine for offline browsing, research, or personal archive. Spidersoft does not aggregate or transmit this data to any central server, nor is it used for search indexing, AI training, or analytics. The tool is purely a client-side utility.

⚙️ Rate Limiting Policy

WebZip is rate-limited because its multi-threaded, high-frequency downloads can overwhelm small to medium web servers, effectively mimicking a low-level denial-of-service attack. Threshold-based blocking (e.g., limiting requests per second per IP) is the recommended countermeasure, as the crawler does not respond to crawl-delay directives and cannot be trusted to self-regulate.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.