🤖 Overview

Cyberspyder is a legitimate web crawler operated by the cybersecurity research firm Cyberspyder Inc., first documented in 2019. Its primary purpose is to systematically scan publicly accessible web applications and Internet-facing services to collect metadata for vulnerability assessment, attack surface mapping, and threat intelligence aggregation. The data feeds into Cyberspyder’s proprietary platform, which provides organizations with continuous external attack surface monitoring and exposure alerts. Unlike malicious scrapers, Cyberspyder is fully transparent, publishing its user‑agent, IP ranges, and operational policies on its official website at cyberspyder.com/crawler.

🌐 Technical Behavior

Cyberspyder performs a combination of shallow and deep crawls, typically starting with a high‑level site map discovery followed by targeted scans of common endpoints such as /robots.txt, /.well-known/, /admin/, and /api/docs. It sends HTTP/1.1 requests with a default rate of one request every three seconds per host, but it can burst up to five requests per second during initial reconnaissance. The bot uses a rotating pool of IPv4 and IPv6 addresses, all registered under ASN AS21629 (Cyberspyder), with ranges documented in the RIPE and ARIN databases. It identifies itself via the User-Agent header — see Detection Indicators — and also sends a custom X-Cyberspyder: 1.0 header on all requests to assist with identification. Cyberspyder respects the Connection: close hint and does not maintain persistent HTTP/2 sessions.

📋 robots.txt Compliance

Cyberspyder fully honors robots.txt Disallow directives, as confirmed by its own documentation and independent testing by the OWASP Crawler Detection Project. It checks the /robots.txt file before each crawl session and caches the rules for 24 hours. If a site returns a 403 or 404 on /robots.txt, Cyberspyder defaults to crawling only the root path and stops after one level of links. The bot also respects the Crawl‑Delay directive when specified.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Cyberspyder/1.0; +https://cyberspyder.com/crawler). A secondary variant — Cyberspyder/1.0 (Windows NT 10.0; Win64; x64) — has been observed on older scans. Behavioral fingerprints include a fixed request interval of three seconds (unless overridden by robots.txt), the presence of the X-Cyberspyder header, and a lack of support for JavaScript or CSS loading. All requests use the GET method and accept text/html, application/xhtml+xml, and application/xml.

📊 Data Usage

Collected data — including server headers, directory structures, SSL certificate metadata, and publicly exposed files — is used exclusively to build and maintain Cyberspyder’s attack surface management platform. The platform helps customers identify misconfigurations, outdated software, and exposed sensitive endpoints. No collected content is used for AI training, advertising, or search indexing; the data is stored encrypted and retained for a maximum of 90 days per Cyberspyder’s published privacy policy.

⚙️ Rate Limiting Policy

Although Cyberspyder is legitimate, its aggressive scanning of internal endpoints (e.g., /wp-admin, /backup) can trigger false positives in WAFs. Rate limiting with a threshold of five requests per second per IP is recommended to prevent unintended resource exhaustion while still allowing the bot to complete its security assessment within a reasonable time window.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.