🤖 Overview

Visbot is a web crawler operated by VisAI Technologies, a company specializing in visual search and image recognition. First detected in web server logs around 2021, the bot systematically collects publicly accessible images, videos, and associated metadata to train the company's proprietary deep learning models for content-based image retrieval. The product it feeds data into is the VisAI Search engine, which indexes visual content without relying on text tags.

🌐 Technical Behavior

Visbot utilizes a distributed crawling infrastructure hosted on Amazon Web Services (AWS) and Google Cloud Platform, with IP addresses typically within the 54.240.0.0/12 and 35.191.0.0/16 ranges. Observed request rates average 40 requests per second with bursts up to 60 RPS during peak indexing. The bot uses HTTP/1.1 persistent connections and a breadth-first crawl strategy, prioritizing pages with high image density and minimal JavaScript dependencies. It sends a custom HTTP header X-Visbot: true on every request. Documentation indicates it follows standard robots exclusion, but independent monitoring shows occasional non-compliance with rapid changes.

📋 robots.txt Compliance

According to the official Visbot documentation at visbot.com/robots, the crawler fully respects Disallow directives and re-fetches robots.txt every 12 hours. However, webmaster community reports indicate that the bot may continue crawling disallowed paths for up to 24 hours after a robots.txt update. The bot also honors Crawl-Delay directives but defaults to a delay of 1 second if unspecified.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Visbot/1.0; +https://visbot.com/bot), with a fallback of Visbot/1.0. Additional identifying headers include X-Visbot: true and a Referer header set to "https://visbot.com". IP addresses typically resolve to hostnames with patterns like "ec2-54-240-xxx-xxx.compute.amazonaws.com".

📊 Data Usage

Collected visual data — including JPEG, PNG, GIF, and WebP files — along with surrounding alt text, captions, and page titles, is used to train convolutional neural networks for object detection and visual similarity. The processed feature vectors are stored in vector databases for rapid querying. PII is intentionally avoided; any unintentionally collected personal data is anonymized and discarded within 30 days.

⚙️ Rate Limiting Policy

Visbot is rate-limited because its aggressive crawling consumes significant server resources, especially on shared hosting platforms. A common recommended threshold is 100 requests per minute per source IP, after which administrators may implement temporary blocks or request rate reduction via the bot's official contact page to ensure fair access for all users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.