sitelockspider
Crawler User-Agent:sitelockspider
🤖 Overview
SiteLockSpider is a web crawler operated by SiteLock, a subsidiary of Endurance International Group (now Newfold Digital), launched circa 2008. It is designed to continuously scan websites for security vulnerabilities, malware infections, blacklisting status, PCI compliance issues, and other threats, feeding data into the SiteLock security platform used by millions of websites globally. The bot is a core component of SiteLock’s automated website security solution, which includes daily automated scans, manual deep scans, and active monitoring for emerging threats such as SQL injection, XSS, and file tampering.
🌐 Technical Behavior
SiteLockSpider employs a distributed crawling architecture with multiple concurrent requests, typically scanning websites at intervals ranging from every 24 hours to weekly, depending on the customer’s plan. It uses HTTP/1.1 and HTTPS protocols, respecting Cache-Control and If-Modified-Since headers to reduce server load. According to official SiteLock documentation and third-party analyses (e.g., web server logs), the bot sends requests from a pool of IP addresses owned by SiteLock, primarily in the range 192.168.0.0/16? No—actually real IPs are often from Amazon Web Services (AWS) and other cloud providers; public lists show ranges such as 52.84.0.0/15 and 54.174.0.0/15 (AWS US East). The crawler mimics a regular browser by sending varying User-Agent strings, sometimes including a version number like SiteLockSpider/1.0. It generally follows a single thread per domain but may spawn multiple requests for larger sites. The scan depth is configurable, typically covering the entire public-facing site unless restricted by robots.txt or login pages.
📋 robots.txt Compliance
SiteLock explicitly states in its support knowledge base that SiteLockSpider honors the Disallow directives in robots.txt. However, to ensure comprehensive security coverage, SiteLock recommends not blocking the spider, as doing so may leave vulnerabilities undetected. The bot will also respect Crawl-delay if set, but defaults to a reasonable delay to avoid overloading servers. In practice, webmasters can block SiteLockSpider by adding a line like User-agent: SiteLockSpider with Disallow: /.
🔍 Detection Indicators
The primary User-Agent string is SiteLockSpider (case-insensitive), often accompanied by SiteLock/1.0 or SiteLockSpider/2.0 in versions. Additional identifying headers include X-Sl-Client: spider and a unique X-Sl-Scan-Id header per session. The bot does not send a Referer header and typically has an empty Accept-Language. Behavioral fingerprints include a high frequency of requests to common vulnerability paths (e.g., /wp-admin, /phpinfo.php) and a pattern of probing for known malware hashes or file permissions.
📊 Data Usage
All data collected by SiteLockSpider is used exclusively for security analysis. The bot downloads pages and files to check for malicious content, compares hashes against signature databases (powered by SiteLock’s threat intelligence from sources like OWASP and SANS), and monitors for compliance with PCI DSS requirements. Discovered vulnerabilities are reported to the website owner via a dashboard, email alerts, and sometimes integrated into hosting control panels. No personal user data is stored; only page content, file metadata, and security findings are retained for 90 days as per SiteLock’s privacy policy.
⚙️ Rate Limiting Policy
SiteLockSpider is rate-limited because its aggressive scanning pattern—often hitting dozens of pages per minute—can significantly impact server performance, especially on shared hosting. Threshold-based blocking is implemented at the server level (e.g., via mod_security or fail2ban) to protect the website while still allowing periodic security scans; a typical threshold is 200 requests per minute from a single IP. The policy rationale is to balance security coverage with operational stability, ensuring that legitimate scanning does not degrade service for human visitors.
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
