🤖 Overview

Webbul-bot is a web crawler operated by Webbul Ltd., a UK-based company specializing in AI-driven web scraping and data extraction services. According to the official Webbul website (webbul.com), the bot is designed to collect publicly accessible web content on behalf of enterprise clients for use in market research, price monitoring, and training proprietary AI models. The crawler feeds data into Webbul’s Data Pipeline Platform, a cloud-based service that normalizes and structures scraped data for analytics and machine learning workflows. Unlike search engine bots, webbul-bot operates on a per-customer basis, scraping specific domains or categories as requested.

🌐 Technical Behavior

Webbul-bot employs distributed crawling using a pool of IP addresses across multiple cloud providers, primarily AWS EC2 and Google Cloud, with ranges documented by the company in their developer portal. The bot sends HTTP/1.1 requests with a default crawl rate of 10 requests per second per IP, but this can increase to 30 requests per second for expedited jobs. It respects Accept headers for HTML, JSON, and XML content, and uses If-Modified-Since headers to reduce redundant downloads. The crawler follows a breadth-first traversal pattern, prioritizing internal links up to a configurable depth (default 3). Webbul documentation notes that the bot may retry failed requests up to three times with exponential backoff (starting at 5 seconds).

📋 robots.txt Compliance

Based on Webbul’s official robots.txt policy published at webbul.com/robots-txt-compliance, the webbul-bot reads and honors all Disallow directives before initiating a crawl job. The company states that “Webbul-bot will never ignore a robots.txt exclusion unless explicitly overridden by a customer with a signed data-use agreement.” In practice, the bot fetches the robots.txt file once per domain at the start of a session and caches it for 24 hours.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Webbul-bot/2.0; +https://webbul.com/bot), though variations with version numbers (1.x, 3.0) have been observed. The bot also includes a custom HTTP header X-Webbul-ClientID that contains a unique customer identifier. Behavioral fingerprints include a consistent request interval of 100ms ±20ms between page loads and a preference for HTTPS over HTTP when available.

📊 Data Usage

Collected data is processed through Webbul’s Extractify pipeline, which applies natural language processing and computer vision models to structure raw HTML. Enterprise clients use the output for competitive intelligence, trend analysis, and training domain-specific AI models. Webbul’s privacy policy states that personal data detected during scraping is automatically redacted.

⚙️ Rate Limiting Policy

Because webbul-bot can sustain high request rates—up to 30 req/s per job—it is reasonable for webmasters to apply rate limiting thresholds (e.g., 100 requests per minute per IP) to protect server performance. Webbul itself encourages customers to set a crawl delay of at least 1 second in their robots.txt for small servers, acknowledging that aggressive scraping may degrade service for human users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.