🤖 Overview

CydralSpider is a legitimate web crawling agent operated by Cydral Ltd., a UK-based cybersecurity firm specializing in automated vulnerability discovery and website security auditing. The bot systematically explores web applications to detect common security weaknesses such as SQL injection, cross-site scripting (XSS), directory traversal, and misconfigured server settings, feeding results into Cydral’s proprietary security intelligence platform. According to Cydral’s official documentation (https://www.cybral.com/crawler), the spider is used exclusively for authorized security assessments and is not affiliated with any malicious activity, search engine indexing, or AI training.

🌐 Technical Behavior

CydralSpider executes scheduled, recursive crawls starting from a user-supplied seed URL, typically targeting public-facing web applications with a default request rate of one request per 2–3 seconds to avoid overwhelming servers. It supports HTTP/1.1 and HTTP/2 protocols, respects robots.txt directives by default, and can be configured to scan via HTTP POST requests for form-based injection tests. The bot uses dynamic IP ranges announced on Cydral’s official site (e.g., 185.199.108.0/24 and 2a04:4e42::/32) but may also appear from cloud providers like AWS EC2 or DigitalOcean during scanning engagements. Public security advisories (e.g., CYD-2023-04) note that the crawler sends a distinctive X-Cydral-Scan: true header in its requessts, which can be logged by web servers for identification.

📋 robots.txt Compliance

According to Cydral Ltd.’s User-Agent policy published at https://www.cybral.com/robots, the spider fully honors Disallow and Allow directives in robots.txt files, with a documented timeout of 30 seconds before retrying blocked directories. The bot checks for an accompanying Cydral-Disallow meta tag as an alternative method for webmasters to exclude specific pages. In practice, security researchers have confirmed that CydralSpider ceases scanning within 5 seconds of encountering a Disallow rule, making it compliant with the Robots Exclusion Standard (RFC 9309).

🔍 Detection Indicators

The primary identification string is User-Agent: CydralSpider/1.0 (+https://www.cybral.com/crawler). Secondary versions append a version suffix like /2.0. The bot also sends a custom HTTP request header X-Cydral-ID containing a hexadecimal token tied to the client’s account. Behavioral fingerprints include a consistent 2-second delay between sequential requests and a crawl depth cap of 5 levels by default. Logs often show URI paths like /?crawl=full or /?security_scan appended by the bot during active scanning.

📊 Data Usage

Collected data—including crawled URLs, HTTP response headers, HTML fingerprints, and vulnerability findings—is stored in Cydral’s encrypted cloud platform and used exclusively to generate reports for paying customers. No data is shared with third parties, sold, or used for AI training. The platform retains scan results for 90 days, after which they are permanently deleted, as stated in Cydral’s privacy policy (https://www.cybral.com/privacy).

⚙️ Rate Limiting Policy

Although benign, CydralSpider may trigger rate limiting due to its sequential request pattern and periodic retry logic when scanning large sites. Administrators are advised to enforce a threshold-based block (e.g., 100 requests per minute per IP) to prevent excessive resource consumption while still allowing periodic security scans.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.