🤖 Overview

WebPix is a legitimate web crawler operated by WebPix Technologies (webpix.com), a company specializing in visual website monitoring and screenshot capture services. Its primary purpose is to periodically fetch web pages and render them as images to detect visual regressions, layout changes, or uptime issues for client websites. The crawler feeds data into the WebPix dashboard, which provides users with timestamped screenshots and visual diff comparisons.

🌐 Technical Behavior

WebPix crawls using a headless Chromium browser to render pages fully, including JavaScript and CSS, before capturing screenshots. It typically requests a single URL per crawl session with a controlled interval between 5 to 60 minutes depending on user configuration. The bot uses IPv4 addresses drawn from a dedicated pool registered to WebPix Technologies, with reverse DNS entries matching *.webpix.com. It supports both HTTP/1.1 and HTTP/2 and sends a unique request ID in custom headers for traceability. According to WebPix documentation, the crawler processes robots.txt before each request and waits for the Content-Length header to avoid loading large files unnecessarily.

📋 robots.txt Compliance

WebPix fully honors robots.txt directives as documented in its official user-agent page at webpix.com/robots. It respects Disallow rules and also supports the Crawl-Delay directive to throttle request frequency. WebPix will not crawl any page explicitly disallowed, and it does not bypass blocked paths via URL manipulation.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; WebPix/1.0; +https://webpix.com/bot). Secondary identifiers include the custom HTTP header X-WebPix-Request-Id and a User-Agent containing “WebPix”. The bot also sets a Referer header to the target URL and includes Accept: image/webp,image/png to indicate screenshot capture preferences.

📊 Data Usage

Collected screenshots and page metadata are used exclusively for visual monitoring and regression detection within the WebPix platform. No data is used for AI training or search indexing. Customers receive visual diffs and historical snapshots, but the raw HTML or content is not stored permanently beyond the screenshot generation window.

⚙️ Rate Limiting Policy

WebPix is rate-limited because its headless browser renders consume significant server resources, and without throttling a single misconfigured client could generate dozens of concurrent requests. A threshold-based block of, for example, 10 requests per second from the same IP within a 60-second window is recommended to protect server stability.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.