🤖 Overview

V-BOT is a legitimate web crawler operated by Vidori Analytics, a company specializing in web data aggregation and AI model training. First documented in 2021, the bot is designed to collect publicly accessible text and metadata from websites to feed into Vidori’s proprietary natural language processing and recommendation systems. Its primary product, the Vidori Insights Platform, uses this data to generate trend analysis and content summaries for enterprise clients. According to the official V-BOT documentation (available at vidori.com/v-bot), the crawler is specifically intended to index high-quality, authoritative sources and respects standard web protocols. It is not associated with any malicious activity.

🌐 Technical Behavior

V-BOT performs breadth-first crawls starting from a seed list of trusted domains, with a default crawl depth of five links. It sends HTTP/1.1 requests with a User-Agent string of V-Bot/1.0 (+http://vidori.com/v-bot) and a secondary identifier Mozilla/5.0 (compatible; V-Bot/1.0; +http://vidori.com/v-bot). Request frequency averages one request every 3–5 seconds per domain, though this can increase to one request per second during burst periods. The bot uses a distributed IP pool registered to Vidori Technologies (ASN 20473) and the Amazon Web Services (ASN 16509) ranges, specifically 52.0.0.0/11 and 54.0.0.0/8. Requests are made over both HTTP and HTTPS, and the bot follows redirects up to five hops. It does not execute JavaScript or parse CSS files; only static HTML and structured data (JSON‑LD, Microdata) are extracted. The crawler operates 24/7 with a cooldown period of 60 seconds if it receives HTTP 429 Too Many Requests responses.

📋 robots.txt Compliance

V-BOT fully honors Disallow directives found in a site’s robots.txt file, as stated in its official documentation and confirmed by technical analysis published by the Web Crawler Ethics Project (github.com/wcethics/v-bot-compliance). It checks robots.txt at least once per crawl session and respects the crawl delay directive (Crawl-Delay) with a minimum of 5 seconds. Evidence from independent audits shows that V-BOT does not circumvent blocked paths or ignore explicit exclusions.

🔍 Detection Indicators

Identifying V-BOT is straightforward via its fixed User-Agent string: V-Bot/1.0 (+http://vidori.com/v-bot). It also sends a custom X-VBOT-Version header set to 1.0. Behavioral fingerprints include a consistent Inter‑Request Interval of no less than 3 seconds and a pattern of requesting robots.txt before the first page of a domain. Log entries show requests with Accept headers containing text/html,application/xhtml+xml and Accept-Language set to en‑US,en;q=0.9.

📊 Data Usage

Collected content is used to train Vidori’s large language models and to power its real‑time trend detection dashboard. Extracted text is tokenized, deduplicated, and stored in a vector database for semantic search. The company explicitly states that personal data (email addresses, phone numbers) is stripped before ingestion, and raw page copies are retained for a maximum of 30 days. No user behavioral data or cookies are collected.

⚙️ Rate Limiting Policy

V-BOT is rate‑limited not because it is malicious, but because its broad crawl scope and high request volume can strain server resources on smaller sites. The recommendation (from Vidori’s own administration guide) is to apply a per‑IP rate limit of 10 requests per minute using standard web‑server modules, with a 429 response after exceeding that threshold to trigger the bot’s built‑in backoff mechanism. This ensures fair access for all crawlers without blocking V-BOT’s legitimate indexing activity.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.