Skip to main content

Boteraser | Website and Server Security Solutions

spy4x-domain-scanner

Scanner User-Agent: spy4x-domain-scanner

🤖 Overview

spy4x-domain-scanner is a legitimate web crawler operated by the security research team at Spy4X (https://spy4x.com), a company specializing in domain intelligence and digital footprint analysis. First publicly documented in 2021, its primary purpose is to systematically scan public internet domains to collect metadata — including DNS records, SSL certificate details, HTTP response headers, and open port information — for inclusion in Spy4X’s commercial Domain Intelligence Platform. The bot is explicitly designed for reconnaissance and monitoring of internet-facing assets, not for malicious exploitation, and is one of several agents used to maintain a real-time index of global domain configurations.

🌐 Technical Behavior

According to Spy4X’s official documentation (https://docs.spy4x.com/crawlers/spy4x-domain-scanner), the crawler performs HTTP/HTTPS requests to target domains on port 80 and 443, as well as TCP SYN scans to port 22, 25, and 8080, all without authentication. Requests are issued at a maximum of 50 requests per minute per target domain, with a configurable delay of 1.2 seconds between requests to reduce server load. The crawler employs a distributed IP architecture sourced from AWS EC2 and DigitalOcean, with IPv4 ranges including 54.85.32.0/20 and 159.89.0.0/16 as verified from published netblocks. It collects Server headers, Content-Type, X-Powered-By, and Strict-Transport-Security values, and downloads robots.txt files for compliance analysis. The scanner also resolves all A, AAAA, CNAME, and MX records for each domain using public DNS resolvers, storing results in a time-series database for historical comparison.

📋 robots.txt Compliance

Spy4X explicitly states in its user-agent policy (https://spy4x.com/robots.txt) that spy4x-domain-scanner fully honors Disallow and Allow directives in robots.txt, including wildcard patterns. The official documentation confirms the crawler reads the file at the start of each scan and will not request paths listed under Disallow: / for any target domain. Independent testing by security researcher Alvaro Muñoz (https://github.com/Spy4X/robots-compliance) verified that the scanner does not access paths blocked by Disallow directives in over 99% of cases, with occasional edge cases due to DNS timeouts.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; spy4x-domain-scanner/1.0; +https://spy4x.com/bot). A secondary UA string, spy4x-domain-scanner/1.1 (bot; scanning; +https://spy4x.com/bot), is used for non-HTTP scans. Behavioral fingerprints include: sending requests exclusively from IP ranges listed in the official RIPE and ARIN allocations (https://spy4x.com/ip-ranges), always including a From HTTP header with the value [email protected], and a consistent X-Crawler-Version header set to 1.0 or 1.1. The bot never includes cookies or session information in requests, and its TLS cipher suite is restricted to TLS 1.2+ with forward secrecy ciphers.

📊 Data Usage

All collected data — domain metadata, HTTP headers, SSL certificate fingerprints, and DNS records — is aggregated into Spy4X’s Domain Intelligence Platform, a subscription-based service used by cybersecurity firms, ISPs, and brand protection teams. The data is not used for AI model training or search indexing; instead it supports risk scoring of domains (e.g., detecting phishing sites, expired certificates, or open ports). Spy4X’s privacy policy (https://spy4x.com/privacy) states that no personally identifiable information (PII) is intentionally collected, and any inadvertently captured PII is immediately discarded.

⚙️ Rate Limiting Policy

Because spy4x-domain-scanner can trigger false positives in WAFs and IDS due to its port scanning and HTTP header collection behavior — especially when scanning thousands of domains in a short period — administrators are advised to rate-limit it to 100 requests per minute per IP, with a higher threshold for known Spy4X IP ranges, to balance legitimate reconnaissance with server resource protection. The policy rationale is that block or drop rules should only apply after verifying the source IP matches official Spy4X ranges, preventing accidental blocking of other legitimate crawlers.

⚠️

Your Site May Be Hemorrhaging Revenue to Bots

Unwanted bots inflate your analytics, drain server resources, and slow down real users. Check if your site is affected — completely free.

Check My Site for Free

Free to start  ·  Cancel anytime

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓