🤖 Overview

Sitevigil is a legitimate web crawler operated by Sitevigil Inc., a company specializing in website uptime and performance monitoring. First documented in 2018, the bot is designed to periodically visit pages to verify availability, measure response times, and detect content changes for alerting purposes. It feeds data into the Sitevigil dashboard and API, used by thousands of webmasters for proactive site reliability management.

🌐 Technical Behavior

The crawler uses HTTP/1.1 with GET requests, typically at intervals configurable by the user (commonly 1–5 minutes for critical sites). Requests are made from IPv4 addresses in the range 104.28.0.0/16 (Cloudflare’s AS13335) and IPv6 blocks 2606:4700::/32. Sitevigil sets the User-Agent header to Sitevigil/3.0 and includes a X-Sitevigil-Origin header with the monitor ID. It does not follow redirects unless explicitly configured, and it honors Cache-Control headers to avoid stale content. The bot sends a single request per target per check cycle, with no concurrent crawling of the same domain.

📋 robots.txt Compliance

According to Sitevigil official documentation (help.sitevigil.com/robots), the bot fully respects Disallow directives in robots.txt. It checks the file before each crawl and caches it for up to one hour. Crawl-delay is also honored when specified, allowing operators to throttle requests per domain.

🔍 Detection Indicators

User-Agent strings observed: Mozilla/5.0 (compatible; Sitevigil/3.0; +https://sitevigil.com/bot). Behavioral fingerprint: exact interval-based requests with IPs from Cloudflare range, and a X-Sitevigil-ID header (e.g., 0a1b2c3d4e5f). Reverse DNS shows crawler.sitevigil.com.

📊 Data Usage

Collected data includes HTTP status codes, response times (in milliseconds), page size, and content hash for change detection. This information is stored in Sitevigil’s cloud backend for up to 90 days and used for uptime alerts, performance graphs, and SLA reporting. No content is scraped or used for AI training.

⚙️ Rate Limiting Policy

Sitevigil is rate-limited because its periodic polling can generate high request volumes on small websites. A threshold of 10 requests per second per IP is recommended to prevent false positives while allowing legitimate monitoring to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.