🤖 Overview

Web Pix is a web crawler operated by WebPix Technologies, a data‑services company headquartered in London, UK. First publicly documented in a 2022 technical whitepaper, the bot systematically harvests publicly available web content—primarily images, PDFs, and structured metadata—to feed into the company’s proprietary Visual AI Training Platform, used by enterprises for computer‑vision model development. Unlike general search‑engine crawlers, Web Pix focuses on visual data and associated context, enabling downstream tasks such as object detection, scene classification, and optical character recognition.

🌐 Technical Behavior

Web Pix employs a distributed crawling architecture that issues HTTP/1.1 GET requests at a steady rate of 10 requests per second per source IP, with spikes up to 20 rps during initial site discovery. The crawler respects conditional GET (If‑Modified‑Since and ETag headers) to avoid redundant downloads. IP ranges are drawn from Amazon Web Services (AWS) EC2 (us‑east‑1, eu‑west‑2) and Google Cloud Platform (us‑central1). The bot’s user‑agent string advertises a default crawl delay of 5 seconds, though observed behavior shows it may ignore Crawl‑Delay directives in robots.txt unless explicitly set to a value ≥2. Web Pix uses a custom URL‑frontier algorithm that prioritises pages containing <img> elements with descriptive alt attributes and structured data (JSON‑LD, schema.org). It does not execute JavaScript or submit forms, relying solely on static HTML parsing.

📋 robots.txt Compliance

According to the official WebPix documentation hosted at webpix.com/crawler-policy, the bot honours all Disallow directives in robots.txt and will not crawl paths explicitly blocked. However, a 2023 community audit on GitHub (repository webpix-crawler-behavior) reported that Web Pix occasionally disregards wildcard patterns (e.g., Disallow: /private/*) if the page is referenced from a high‑authority external link. The operator has since patched this in version 1.4.2. For sites without a robots.txt file, Web Pix assumes full crawl permission.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; WebPix/1.0; +http://www.webpix.com/bot). Secondary variants append +ImageCollector or +PDFHunter for specialised tasks. Web Pix always includes the request header X‑WebPix‑Crawl‑ID: UUID, where UUID is a unique crawl session identifier. Behavioural fingerprints include consistent 5‑second intervals between requests and a distinct preference for fetching .jpg, .png, .gif, and .pdf files first, before any HTML content.

📊 Data Usage

Collected images and metadata are processed through WebPix’s AutoLabel pipeline, which applies automated annotation (bounding boxes, captions) and feeds the results into the company’s visual AI training sets. The data is also used to generate synthetic training examples via generative adversarial networks (GANs) for edge‑case scenarios. WebPix does not sell raw data publicly, but licenses aggregated, anonymised visual‑trend reports to AI research labs and e‑commerce analytics firms.

⚙️ Rate Limiting Policy

Rate‑limiting Web Pix to 10 requests per minute per IP is recommended because the bot’s sustained 10‑rps crawl can overwhelm small‑to‑medium web servers and degrade performance for human visitors. The policy is not a block—threshold‑based limits simply throttle the crawler to a safe level, preserving site availability while still allowing collection of needed visual data.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.