🤖 Overview

Wotbox is a web crawler operated by WOT Services Ltd., the company behind the Web of Trust (WOT) browser extension and website reputation service. Its primary purpose is to automatically scan publicly accessible websites to collect content, user reviews, and metadata that feed into WOT’s trustworthiness scoring algorithms, helping users avoid malicious or fraudulent sites. The crawler was first documented around 2007 and has been continuously updated to support modern web standards.

🌐 Technical Behavior

Wotbox performs periodic, distributed crawls of URLs submitted by the WOT community and new domains discovered through link analysis. It uses IP ranges such as 91.199.212.0/24 and 91.199.213.0/24, as listed in security advisories and WHOIS data. The crawler sends HTTP GET requests at a rate of approximately one request every two to four seconds per individual IP, respecting Crawl-Delay directives in robots.txt. It uses HTTP/1.1 with keep-alive connections and does not execute JavaScript, relying entirely on raw HTML and metadata extraction. Wotbox also fetches common resource files (like robots.txt and sitemaps) to determine crawl boundaries and avoid overloading servers.

📋 robots.txt Compliance

According to WOT’s official crawler policy page at https://www.mywot.com/en/crawler, Wotbox fully honors robots.txt exclusion rules and will immediately cease crawling any path disallowed by a Disallow directive. Site operators have verified through server logs that the crawler checks robots.txt on each visit and respects changes within minutes, though some have noted a short grace period before full compliance after a rule update.

🔍 Detection Indicators

The primary User-Agent string is Wotbox/2.0 (+http://www.mywot.com/wotbox), with variants including Wotbox/2.0 (bot; http://www.mywot.com/bot) and Mozilla/5.0 (compatible; Wotbox/2.0). The crawler often includes a Via header identifying a proxy and a From header with the crawler contact email. Behavioral fingerprints include a consistent request interval, no JavaScript execution, and a preference for text/html content types. Security researchers have also observed that Wotbox rarely sends cookies or HTTP authentication.

📊 Data Usage

Data collected by Wotbox is aggregated into WOT’s reputation database, which powers real-time safety ratings displayed in the WOT browser extension (available for Chrome, Firefox, and Edge). The system combines automated analysis of page structure and content with user-submitted reviews to classify domains as trusted, suspicious, or malicious. WOT API customers (including antivirus vendors and security platforms) also use this data for threat intelligence feeds. The crawler does not store personal information, only public web content and metadata.

⚙️ Rate Limiting Policy

Wotbox is rate-limited because its distributed, ongoing scans can generate significant load on servers, especially for sites with high page counts or limited resources. A threshold-based policy—such as limiting requests to 1 per second per IP or blocking after 50 requests per minute—is recommended to prevent resource exhaustion while still allowing the legitimate crawler to complete its periodic evaluations.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.