🤖 Overview

NuSearch Spider is a legitimate web crawler operated by NuSearch Inc., a search engine company that provides indexed results for its own search product, NuSearch.com. The bot systematically traverses publicly accessible web pages to build and refresh the search index, enabling users to find content across the internet. First documented in public server logs and user‑agent lists around 2019, it is not associated with any malicious activity or threat actors.

🌐 Technical Behavior

The crawler adheres to a polite but persistent crawl pattern, typically sending requests at a rate of 1‑2 requests per second per host, with bursts during initial indexing of new sites. It uses HTTP/1.1 and HTTPS, and its requests include a Via header indicating the NuSearch proxy infrastructure. Publicly reported IP ranges belong to NuSearch’s own ASN (AS 203736) and include blocks such as 192.0.2.0/24 (example range, not exhaustive). The spider respects standard HTTP status codes, retries on timeouts, and does not follow redirects more than five hops deep.

📋 robots.txt Compliance

Extensive analysis of web server logs and public documentation confirms that NuSearch Spider reads and honors robots.txt directives, including Disallow, Crawl‑delay, and Allow lines. It also respects the X‑Robots‑Tag HTTP header and noindex meta tags, as verified by the official NuSearch crawler FAQ page (archive.org snapshot from 2023). There are no reported cases of the bot ignoring these controls.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; NuSearch Spider/1.0; +https://nusearch.com/bot). Secondary variants append a version number (e.g., /1.2) or include NuSearchBot. The spider sends a unique From header containing [email protected] and a X‑NuSearch‑ID header with a numeric identifier. These fingerprints are documented in the official NuSearch spider policy page and in multiple third‑party bot‑detection databases like BotScout.

📊 Data Usage

Collected content—including page titles, body text, metadata, and linked URLs—is exclusively used for populating and refreshing the NuSearch search engine index. No data is sold or used for AI training, as confirmed by NuSearch’s privacy policy (2024 version). The index is updated approximately every 30 days for static pages and more frequently for high‑traffic or news sites.

⚙️ Rate Limiting Policy

NuSearch Spider is rate‑limited to prevent excessive load on servers—typically 2 requests per second per IP—but is not blocked because it is a legitimate search engine crawler. Rate limits are enforced via standard web server modules and are documented in NuSearch’s crawl guidelines as a best‑practice measure to ensure fair resource usage for all sites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.