🤖 Overview

dtsearchspider is a web crawler developed and operated by dtSearch Corporation, a company specializing in enterprise text retrieval and indexing software. The spider is a component of the dtSearch product line, purpose-built to crawl public and internal websites and collect document contents for indexing into the dtSearch text‑retrieval engine. Unlike general‑purpose search engine crawlers, dtsearchspider is deployed by individual organizations (or by dtSearch itself for demonstration and support purposes) to create searchable archives of web‑accessible content. Its primary goal is to feed the dtSearch index, enabling fast full‑text queries across large document corpora. Official product documentation (dtsearch.com) describes the spider as an optional module available in dtSearch Desktop, dtSearch Web, and dtSearch Network editions, first introduced in the early 2000s and continuously updated.

🌐 Technical Behavior

dtsearchspider operates as a multi‑threaded crawler that follows hyperlinks, downloads supported document formats (HTML, PDF, Microsoft Office, plain text, and others), and stores extracted text for indexing. Default crawl behaviour includes a configurable delay between requests—typically 1‑5 seconds—but users can override this, potentially causing high request rates. The bot uses standard HTTP/1.1 GET requests and honors If‑Modified‑Since and ETag headers to avoid re‑downloading unchanged documents. IP ranges are not fixed; the spider originates from whichever host runs the dtSearch software, which could be a corporate network or a cloud VM. dtSearch does not publish a dedicated block of IP addresses. The crawler respects noindex meta tags in addition to robots.txt directives. Official documentation warns that aggressive crawl settings can overwhelm servers, and recommends testing against a staging environment before production deployment.

📋 robots.txt Compliance

According to the dtSearch user manual (available at dtsearch.com/support/documentation), dtsearchspider fully honors robots.txt Disallow directives by default. The spider reads the file at the root of each domain before crawling and skips any URL paths declared off‑limits. It also respects Crawl‑delay directives when present, limiting the rate to the specified number of seconds. Users can disable robots.txt parsing via configuration, but the software defaults to compliance. This has been confirmed in multiple security blogs and forum posts (e.g., Stack Overflow threads discussing dtSearch deployment).

🔍 Detection Indicators

The primary detection method is the User‑Agent string, which follows the format dtSearchSpider or dtSearch/Spid (e.g., “dtSearchSpider/1.0” or “dtSearch/8.5.0 Spider”). No additional custom HTTP headers are documented. Behavioural fingerprints include high request concurrency (if misconfigured) and a tendency to request files with extensions like .pdf, .docx, and .txt. The bot does not identify itself with a version-specific string in all cases; older versions may omit the version number. Logs showing repeated access to scripted URLs or dynamic endpoints are indicative of an unconfigured crawler using default settings.

📊 Data Usage

Data collected by dtsearchspider is exclusively used to build and maintain a local text index for the dtSearch retrieval engine. This index supports full‑text search across crawled documents, with features like advanced Boolean queries, fuzzy searching, and thesaurus support. The index is stored on the organization’s own server or workstation; dtSearch Corporation does not access or aggregate the crawled data. Any documents retrieved are stored temporarily during indexing and then retained only to serve search results to authorized users (as configured by the deploying organization). No data is sent to dtSearch’s cloud or used for third‑party analytics or AI training.

⚙️ Rate Limiting Policy

Because dtsearchspider can be configured by end users to send aggressive request rates—sometimes thousands of requests per minute without intentional throttling—web application administrators are advised to rate‑limit it using per‑IP or per‑User‑Agent thresholds. The rationale is not that the bot itself is malicious, but that a misconfigured instance can degrade server performance for other legitimate users. Standard rate‑limiting policies (e.g., 10‑20 requests per second) protect site stability while still allowing the crawler to complete its indexing work within acceptable timeframes, as recommended in the dtSearch documentation.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.