🤖 Overview

HybridBot is a web crawler operated by Hybrid, a company specializing in AI training data and model improvement services. First publicly documented in early 2023, the bot is designed to collect publicly accessible web content for training Hybrid’s proprietary language models and for improving search relevance within its products. According to Hybrid’s official website, the crawler helps “build better AI while respecting website owners’ preferences.” It is not associated with any malicious activity and is classified as a legitimate, rate-limited agent.

🌐 Technical Behavior

HybridBot uses a configurable crawl rate that defaults to a maximum of 10 requests per second per domain, as stated in its official documentation. It follows standard HTTP/1.1 and HTTP/2 protocols, employs a rotating set of IP addresses primarily from AWS and Google Cloud (ranges publicly listed in its DNS records – e.g., 34.x.x.x and 35.x.x.x blocks), and respects Retry-After headers. The crawler typically fetches both HTML and CSS/JS resources to render pages as a modern browser might, but it does not execute embedded JavaScript unless explicitly required for page rendering. HybridBot’s user-agent string is “HybridBot” with optional version suffixes like “HybridBot/1.0”. It identifies itself in the User-Agent header and additionally sends a From header containing a contact email. The bot also includes a X-Robots-Tag handler, obeying noindex directives.

📋 robots.txt Compliance

HybridBot fully complies with the Robots Exclusion Protocol. Official documentation confirms it reads robots.txt at crawl initiation and honors Disallow, Allow, Crawl-Delay, and Sitemap directives. Hybrid also provides a dedicated compliance page (robots.hybridbot.com) that logs when a site’s robots.txt is fetched. There are no known cases of HybridBot ignoring disallow rules; community reports on webmaster forums confirm its good behavior.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: HybridBot/1.0 (or simply HybridBot). Additional headers include From: [email protected] and X-Hybrid-Crawler: 1. The crawler’s IP addresses belong to known cloud provider netblocks; a regularly updated JSON list is available at hybridbot.com/ip-ranges.json. Behavioral fingerprints include a consistent request pattern of fetching a robots.txt first, then sequentially requesting pages from a sitemap if present.

📊 Data Usage

Collected data is used exclusively for training Hybrid’s conversational AI models and improving search relevance in the Hybrid Search platform. According to Hybrid’s privacy page, data is anonymized and aggregated; personal information is stripped. The company does not sell the crawled data to third parties and requires a data processing agreement for any commercial use. HybridBot does not collect cookies or store login sessions.

⚙️ Rate Limiting Policy

HybridBot is rate-limited by default because its high request volume – up to 10 req/s – can cause performance issues on smaller websites. Web administrators are encouraged to set a Crawl-Delay in robots.txt to throttle the bot further; Hybrid’s documentation explicitly supports this and advises that threshold-based blocking (e.g., blocking after 100 requests per minute) is a reasonable policy to prevent resource exhaustion while still allowing the bot to index content.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.