🤖 Overview

The sitecheck crawler is operated by Sucuri Inc., a subsidiary of GoDaddy, as part of their SiteCheck remote website security scanner. Launched in 2010, its primary purpose is to automatically scan publicly accessible web applications for known malware, malicious redirects, defacements, blacklisting status, and outdated software versions. The data feeds into Sucuri’s security dashboard and public reports, enabling website owners and security teams to identify vulnerabilities without installing an agent.

🌐 Technical Behavior

sitecheck initiates HTTP GET requests to the root URL and common paths such as /wp-admin/, /administrator/, /phpMyAdmin/, and /.env to probe for exposure. It also fetches a site’s homepage and a few random pages to detect injected JavaScript or iframes. According to Sucuri’s documentation, the crawler employs a configurable request rate typically between 1 and 5 requests per second, with a default timeout of 30 seconds per page. IP addresses originate from Sucuri’s own anycast network (ranges like 192.124.249.0/24 and 185.93.228.0/24) and from shared cloud providers such as AWS and Linode. The scanner uses HTTP/1.1 with a keep-alive connection and sends a X-Forwarded-For header if provided by the hosting environment. It does not execute JavaScript or render pages; it only parses raw HTML and HTTP headers to identify anomalies.

📋 robots.txt Compliance

Sucuri states that sitecheck honors the robots.txt Disallow directives when they are explicitly set for the user-agent “SiteCheck” or “sitecheck.” However, documentation notes that the scanner may ignore rules that block entire directories if the scan target is the root domain, as the scanner needs to check those paths for security risks. Site owners are advised to use a dedicated user-agent directive (e.g., User-agent: SiteCheck Disallow: /wp-admin/) rather than a catch-all. Official Sucuri documentation (https://sitecheck.sucuri.net/about) confirms this behavior and recommends whitelisting the IP ranges if robots.txt does not suffice.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; SiteCheck/1.0; +https://sitecheck.sucuri.net). Older variants may include SiteCheck/1.0 alone or Sucuri/1.0. Behavioral fingerprints include requests to non-existent common admin paths, a predictable timing pattern of ~200ms between requests, and the absence of Accept-Language or Referer headers. Server logs often show the IPs resolving to hostnames like sitecheck.sucuri.net or sucuri.net. The scanner also sets a custom X-Sucuri-ID header in some cases, documented in Sucuri’s API references.

📊 Data Usage

Collected data—including detected malware signatures, blacklist status from sources like Google Safe Browsing, and version fingerprints—is aggregated into the Sucuri Security Dashboard and used to generate site health reports and alert notifications. No raw page content is stored; only security indicators are retained. The data also trains Sucuri’s machine learning models for new malware detection patterns, as outlined in their privacy policy (https://sucuri.net/privacy).

⚙️ Rate Limiting Policy

sitecheck is rate-limited by Sucuri to a maximum of 5 requests per second per target domain, ensuring it does not overwhelm low-resource servers. However, website operators may impose their own threshold-based blocking (e.g., banning IPs exceeding 10 requests per 10 seconds) for redundant scans, as the tool is intended for brief, periodic checks rather than continuous crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.