🤖 Overview

Woobot is a legitimate web crawler operated by WooRank, a SaaS platform specializing in website analysis, SEO auditing, and competitive benchmarking. First documented in official WooRank documentation, the bot systematically crawls public websites to generate detailed reports on technical SEO, site structure, and performance metrics as part of WooRank’s automated website review service. Unlike malicious scrapers, Woobot is designed solely to provide site owners actionable insights and is explicitly listed in WooRank’s public bot information page (WooRank Bot Page).

🌐 Technical Behavior

Woobot performs regular, scheduled crawls initiated from WooRank’s dedicated server infrastructure, typically using a fixed set of IP ranges announced in their Whois records (e.g., 104.18.x.x or Cloudflare origin IPs). The bot requests HTML pages, CSS, JavaScript, and image resources to simulate a real user’s browsing experience, but it does not interact with forms or submit data. Crawl frequency depends on the site’s update activity and the user’s subscription tier — for free accounts, WooRank may re-crawl every 30 days; for paid plans, weekly or daily. The bot respects HTTP status codes and will back off on 429 or 503 responses, but it does not follow meta robots tags beyond robots.txt directives. Connections are made over both HTTP/1.1 and HTTP/2, with a default request timeout of 30 seconds.

📋 robots.txt Compliance

Yes, Woobot fully honors robots.txt Disallow directives. WooRank’s official documentation explicitly states that the bot respects robots.txt and that site owners can block specific paths or the entire bot by adding User-agent: Woobot and Disallow: / to their robots.txt file. Tests conducted by various site administrators confirm immediate compliance after robots.txt updates.

🔍 Detection Indicators

The primary User-Agent string is Woobot/1.0 (e.g., Mozilla/5.0 (compatible; Woobot/1.0; +https://www.woorank.com/bot)). Additional identifying headers include From: [email protected] and a custom X-Woobot: 1 header on some requests. Behavioral fingerprints include a consistent crawl interval of at least 10 seconds between requests and a tendency to start crawls during off-peak hours in the target site’s time zone.

📊 Data Usage

Data collected by Woobot is used exclusively to populate WooRank’s website analysis reports, which include metrics such as page load speed, meta tag validation, alt text presence, and mobile-friendliness scores. The information is stored on WooRank’s servers and made available to the account holder who initiated the crawl; it is not used for AI training or resold to third parties. WooRank’s privacy policy further limits data retention to the duration of the user’s subscription.

⚙️ Rate Limiting Policy

Woobot is rate-limited because its periodic scans can generate sudden request spikes on small websites, potentially degrading performance for other users. Organizations typically apply threshold-based blocking (e.g., 50 requests per minute per IP) to prevent resource exhaustion while still allowing the bot to complete its legitimate SEO analysis.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.