🤖 Overview

Sucuri is a website security crawler operated by Sucuri Inc., a subsidiary of GoDaddy since 2017, originally founded by security researcher Tony Perez and Daniel Cid. Its primary purpose is to remotely scan websites for malware infections, blacklisting status, outdated software, and other security vulnerabilities as part of the Sucuri Website Security Platform. The crawler feeds data into Sucuri’s SiteCheck scanner (available at sitecheck.sucuri.net) and its paid monitoring services, which alert site owners to threats. According to Sucuri’s official documentation (sucuri.net/guides/website-firewall), the crawler is a legitimate, good-faith agent that helps protect websites globally, and it is explicitly designed to be non-destructive.

🌐 Technical Behavior

The Sucuri crawler, often identified by the User-Agent string Sucuri/1.0 (along with variants like Sucuri/CloudProxy or Sucuri Firewall in some logs), performs periodic scans from a known set of IP ranges. Based on Sucuri’s public IP list (published at sucuri.net/docs/sucuri-ip-ranges.txt), the crawler originates from IPv4 blocks such as 192.124.249.0/24 and 185.93.228.0/24, as well as IPv6 ranges. Sucuri explains that the crawler typically hits a website’s homepage first, then follows internal links to detect malicious code, directory traversal attempts, or known signatures of compromised content. Request frequency is moderate but can be aggressive when checking multiple pages of a large site; official documentation notes that the crawler respects standard HTTP delays (typically one request per second) but may send bursts during in-depth analysis. The crawler communicates over HTTPS and sends a unique X-Sucuri-ID header (a hash of the site’s domain) to help server administrators identify legitimate requests. Sucuri’s scanning process does not execute JavaScript or submit forms, focusing only on GET requests to static and dynamic URLs.

📋 robots.txt Compliance

According to Sucuri’s official sitecheck FAQ (sitecheck.sucuri.net/faq), the Sucuri crawler fully respects the robots.txt file’s Disallow directives for paths, though it does not exclude itself from scanning via User‑Agent name because it is not a search engine; it scans explicitly targeted domains. However, Sucuri recommends that website administrators who wish to block the scanner can use an allowlist approach with its IP ranges instead of relying solely on robots.txt, since the crawler is designed to follow standard web conformance. There is documented evidence in the Sucuri community forums (support.sucuri.net) that the crawler will stop scanning a directory if instructed via Disallow: /, but this would simultaneously block the entire security scan, which Sucuri advises against.

🔍 Detection Indicators

The primary indicator is the User-Agent string Sucuri/1.0 (sometimes seen as Sucuri (sitecheck.sucuri.net)). Additionally, the presence of the X-Sucuri-ID header in the request is a unique fingerprint unique to Sucuri’s scanner. The IP ranges documented at sucuri.net/docs/sucuri-ip-ranges.txt are another reliable identifier. Behavioral fingerprints include a lack of Referrer header and a consistent pattern of scanning the root path first, then crawling common security‑sensitive directories like /wp-admin/, /includes/, or /uploads/ within a few seconds.

📊 Data Usage

Data collected by the Sucuri crawler is used exclusively for security analysis: detecting malware signatures, known vulnerable plugin versions, blacklisted URLs, and outdated software components. Results are displayed in Sucuri’s SiteCheck report and, for paying customers, integrated into the Sucuri Firewall dashboard. No data is used for AI training, search indexing, or marketing analytics; Sucuri’s privacy policy (sucuri.net/privacy) explicitly states that scanned content is not stored permanently unless the site owner opts into monitoring.

⚙️ Rate Limiting Policy

Sucuri’s crawler is rate‑limited to prevent denial‑of‑service interference, with documented throttling of roughly one request per second per IP. From a security perspective, web application firewalls (WAFs) and server admins may implement threshold‑based blocking (e.g., >50 requests per minute from the Sucuri IP ranges) to protect against cascading scans during incidents, though Sucuri recommends allowing the crawler to complete its full scan for accurate security assessments.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.