🤖 Overview

vlsearch is a web crawler operated by Vectara, a company specializing in retrieval-augmented generation (RAG) and neural search infrastructure for enterprise applications. First documented in early 2025, vlsearch is designed to index publicly accessible web content that feeds into Vectara’s Vectara Search Platform, enabling semantic search and conversational AI experiences for customers. Unlike general-purpose search engine bots, vlsearch targets content specifically requested by Vectara’s enterprise clients to power custom RAG pipelines.

🌐 Technical Behavior

According to Vectara’s official documentation and the robots.txt guidance published on their website (vectara.com), vlsearch performs both deep crawling and incremental re-crawls of domains that have opted in through Vectara’s customer dashboard. The crawler respects a crawl delay of at least 10 seconds between requests by default, though this can be configured per customer. vlsearch sends requests over HTTP/1.1 and HTTP/2 from a documented set of IP ranges owned by Amazon Web Services (AWS), specifically in the us-east-1 and eu-west-2 regions. The bot identifies itself via the User-Agent string “VectaraSearchBot/1.0” (also known as “vlsearch”) and typically adds an X-Vectara-Crawl header indicating the customer account ID. The crawler follows links recursively but avoids binary files (e.g., .exe, .zip) unless explicitly allowed. It also respects the noindex meta tag and the X-Robots-Tag HTTP header.

📋 robots.txt Compliance

Vectara publicly states that vlsearch fully respects the robots.txt exclusions and Disallow directives. In testing conducted by the Vectara team, the crawler was verified to honor both user-agent-specific and global rules. The bot checks robots.txt before each crawl session and caches the file for up to 24 hours. Operators can block vlsearch entirely by adding User-agent: VectaraSearchBot followed by Disallow: / to their robots.txt, as documented on the Vectara support page.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: “VectaraSearchBot/1.0” (alias “vlsearch”). Secondary headers include X-Vectara-Crawl containing a customer-specific UUID, and the From header may contain a contact email. Behavioral fingerprints include a consistent request interval of at least 10 seconds and typical AWS IP ranges. Network administrators can monitor logs for these strings to differentiate vlsearch from other bots.

📊 Data Usage

Collected web content is used exclusively to build semantic search indexes and RAG knowledge bases for Vectara’s paying enterprise customers. The data is processed, chunked, and vectorized into Vectara’s proprietary retrieval engine, which powers conversational AI and document search features. Vectara’s privacy policy states that indexed content is not shared with third parties and is retained only for the duration of the customer’s contract.

⚙️ Rate Limiting Policy

Because vlsearch may aggressively re-crawl high-value enterprise content after onboarding, rate limiting is recommended to prevent excessive load. Vectara advises system administrators to enforce threshold-based blocking (e.g., 100 requests per minute per IP) to protect server resources while still allowing legitimate indexing of opted-in domains.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.