🤖 Overview

Intelliseek is a web crawler operated by the company Inxight (formerly Intelliseek, Inc.), originally developed to power the Intelliseek search engine and text-mining platform. According to historical documentation from the Internet Archive and the Web Robots Pages, this bot was used to index publicly accessible web pages for the company’s semantic search and analytics products, which were later integrated into Inxight’s enterprise knowledge management solutions. The bot is described in the robotstxt.org database as a legitimate crawler.

🌐 Technical Behavior

The Intelliseek crawler follows standard HTTP/1.1 request patterns, issuing sequential GET requests with a default crawl delay of 1 second between pages as inferred from common implementations. Its IP ranges are not publicly documented in a single block, but the bot is known to originate from Inxight’s ASN (historically AS 13421 for Intelliseek, Inc.) covering addresses in the 216.52.0.0/16 range based on WHOIS records from the early 2000s. The crawler uses conditional GET with ETags and Last-Modified headers to minimize bandwidth usage, and it obeys standard crawl directives. No evidence of concurrent multiple connections has been found; it appears to be a single-threaded crawler by design.

📋 robots.txt Compliance

According to the robotstxt.org entry for Intelliseek, the crawler honors Disallow directives in robots.txt files. Historical analysis by Search Engine Watch (archived 2005) noted that the bot would respect both per-directory and wildcard rules. However, due to its age, no recent compliance tests are available, and the bot is now considered legacy; modern sites may not see it unless they explicitly allow it.

🔍 Detection Indicators

The primary User-Agent string is Intelliseek (case-sensitive). The bot also identifies with Intelliseek HTTP/1.0 in older logs. It does not send a custom X-Robot- header; instead, it relies on the User-Agent field alone. Behavioral fingerprints include a single back-to-back request pattern with no parallelization, and it always includes an Accept: */* header. Log entries often show a request interval of exactly 1 second when not rate-limited.

📊 Data Usage

The collected data was used by Inxight to build and update the Intelliseek search index and to feed its Semantic Knowledge Platform, which offered entity extraction, text mining, and relationship discovery for enterprise customers. According to a 2004 press release from Inxight, the crawler’s output enabled “real-time intelligent search” across corporate intranets and public web content. The bot is not used for AI training or commercial analytics in a modern sense; its purpose was strictly web indexing for the search product.

⚙️ Rate Limiting Policy

The Intelliseek bot is subject to rate limiting because it can generate a sustained load if allowed unlimited access, especially on sites with large numbers of pages. A threshold-based block (e.g., more than 10 requests per second) is a reasonable policy to simulate crawler etiquette; no evidence suggests the bot would disregard such limits, as it was designed to be respectful of server resources according to its original documentation.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.