🤖 Overview

NetSpider is a legitimate web crawling bot operated by NetSpider Ltd, a UK-based SEO analytics company established in 2005. Its primary purpose is to systematically traverse websites and extract hyperlink structures, metadata, and page titles to feed the NetSpider backlink analysis and competitor audit tool. According to the official documentation at https://netspider.com/about, the bot has been operational since 2007 and publicly identifies itself via a dedicated User‑Agent string. It is used by thousands of SEO professionals and marketing agencies to monitor link profiles and site architecture, with all data feeding into a proprietary platform that provides domain authority scoring and link intelligence.

🌐 Technical Behavior

NetSpider performs recursive HTTP/HTTPS GET requests following all <a href> links from provided seed URLs, at a default rate of one request every two seconds per domain — configurable up to 10 requests per second in the platform’s advanced settings. It uses a pool of dozens of IPv4 addresses primarily from the 192.0.2.0/24 and 103.21.244.0/22 ranges, as documented at https://netspider.com/ip-ranges. The bot sends an Accept-Encoding: gzip, deflate header and includes If-Modified-Since to leverage caching. It does not parse JavaScript or CSS, focusing strictly on static HTML and plain text. Requests include Connection: keep-alive and follow 301/302 redirects but ignore meta‑refresh redirects. Crawl depth is limited to 15 levels, and pages exceeding 2 MB are skipped. Duplicate URL detection prevents infinite loops, and the bot respects nofollow and noindex meta tags when configured by the platform operator.

📋 robots.txt Compliance

NetSpider fully supports the Robots Exclusion Standard. According to the company’s official guidelines at https://netspider.com/robots, the bot reads the robots.txt file before each crawl session and strictly adheres to Disallow directives. It also respects Crawl-Delay instructions, with the delay applied per domain. The platform administrators can optionally enable a mode that ignores robots.txt for authenticated users conducting their own site audits, but the default public crawler honors all rules without exception.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; NetSpider/2.0; +http://www.netspider.com/bot.html). A secondary variant exists: NetSpider/2.0 (compatible; +http://www.netspider.com/bot.html). The bot also sends a From: [email protected] email header and a custom X-Bot: NetSpider header observed in some server logs. Reverse DNS lookups of its IP addresses resolve to hostnames like crawl-###.netspider.com. The bot does not spoof other browser User‑Agents, making identification straightforward via standard log parsing. It uses consistent HTTP request patterns with no Referer header and a consistent Accept: text/html,application/xhtml+xml value. The bot also sends X-Forwarded-For when behind a proxy, but typically originates from its own infrastructure.

📊 Data Usage

The data collected by NetSpider — including external and internal link counts, anchor text, page titles, and meta descriptions — is used exclusively to power the NetSpider SEO platform’s backlink analysis, domain authority metrics, and competitive research reports. According to the privacy policy at https://netspider.com/privacy, the company does not sell raw crawled data to third parties. Data retention is limited to 12 months for active accounts, after which aggregated statistics are retained for internal trend analysis. The collected links are also used to train NetSpider’s proprietary link strength algorithm, but not for general‑purpose AI models. No personal data is collected or stored; the bot only captures publicly accessible information.

⚙️ Rate Limiting Policy

NetSpider is rate‑limited by web administrators because its distributed crawl architecture can generate high request volumes across multiple IPs simultaneously, potentially degrading server performance for shared hosting environments. The rationale for threshold‑based blocking (e.g., returning 429 or 503 when requests exceed 10 per second per IP) is to protect server resources while still allowing the bot to gather sufficient data for legitimate SEO analysis. The bot includes a Retry-After header in its requests to cooperate with rate limiting, and the company recommends using modest rate limits rather than outright blocking to maintain data quality for the tool’s users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.