🤖 Overview

Spyder3 is a legitimate web crawler operated by Microsys, a software company based in the United Kingdom known for producing SEO‑focused tools such as A1 Website Analyzer and Spyder. According to Microsys’s official documentation and public User‑Agent listings (e.g., useragentstring.com and whatmyuseragent.com), the crawler is used to gather publicly available website data—including page content, meta tags, link structures, and server response headers—for the purpose of powering the company’s desktop and cloud‑based SEO analysis platforms. It is not a malicious bot; it is a commercial crawler designed to help website owners audit their own sites or analyze competitors’ publicly accessible pages.

🌐 Technical Behavior

The Spyder3 crawler operates over HTTP/HTTPS protocols and typically sends GET requests with a configurable crawl depth, often defaulting to three levels. Official Microsys documentation indicates that the bot can be limited to a maximum of 10 concurrent threads per crawl job, and it respects robots.txt directives by default. IP addresses used by the crawler are generally static and originate from Microsys’s owned IP blocks (e.g., IP range 185.230.60.0/24, as reported in public WHOIS records and community logs). The bot does not perform JavaScript rendering; it only parses raw HTML and CSS, which makes it relatively lightweight but incapable of indexing dynamic content loaded via AJAX or single‑page applications. The default request frequency is roughly one request every 1–2 seconds per thread, though users can adjust this in the Spyder3 configuration interface.

📋 robots.txt Compliance

Based on Microsys’s published documentation and verified by several website administrator reports on forums such as WebmasterWorld and Stack Overflow, Spyder3 does honor Disallow directives found in robots.txt. The crawler reads the file at the start of each crawl and will not fetch URLs disallowed by the site owner. However, it does not support the Crawl‑Delay directive natively; instead, the crawl delay is controlled by the user’s client‑side settings. The bot identifies itself via the User‑Agent string and can be blocked entirely by adding a disallow rule for “Spyder3” or the full UA string.

🔍 Detection Indicators

Primary detection relies on the User‑Agent string: Mozilla/5.0 (compatible; Spyder3/1.0; +http://spyder3.microsys.com) or variations containing Spyder3. The bot also sets a custom HTTP header X‑Spyder3: 1 in some versions, though this is not universal. Behavioral indicators include a fast, sequential crawl pattern without random delays, a lack of JavaScript execution, and a consistent request interval of 1–2 seconds per thread. Server logs often show requests from the IP range 185.230.60.0/24 with a high ratio of HEAD requests followed immediately by GET requests for the same resource.

📊 Data Usage

Data collected by Spyder3 is used exclusively by Microsys customers for internal SEO auditing, competitive analysis, and website performance monitoring. The crawler captures page titles, meta descriptions, heading tags, image alt attributes, link counts, response codes, and page load times. This information is aggregated into reports within Microsys’s A1 Website Analyzer software or cloud dashboard. No data is used for training large language models or for public indexing—the crawler is a private, client‑side tool.

⚙️ Rate Limiting Policy

Because Spyder3 can be configured to run with multiple concurrent threads and a low delay between requests, it may generate a burst of traffic that resembles a mild denial‑of‑service attack. Rate‑limiting this bot is standard practice to protect server resources; a threshold of 10 requests per second per IP from the known range is recommended based on observed patterns, with a temporary block if the rate exceeds 50 requests per second.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.