🤖 Overview

Zibber-v is a legitimate web crawler operated by Zibber, a company that offers a platform for website monitoring, change detection, and structured data extraction. First documented in public user‑agent lists around 2021, the bot’s primary purpose is to crawl websites on behalf of Zibber’s paying customers, who use the collected data to track competitor pricing, monitor content updates, and feed SEO dashboards. According to Zibber’s official documentation at zibber.com, the crawler is explicitly designed for ethical, consent‑based scraping and does not target sensitive or login‑protected areas.

🌐 Technical Behavior

Zibber‑v performs both broad and focused crawling, typically using HTTP/1.1 and HTTP/2 protocols. It requests pages in rapid succession but respects a default crawl delay, which can be customised per customer. The bot’s IP ranges are sourced from major cloud providers such as AWS (EC2) and Google Cloud Platform, as confirmed by reverse DNS lookups published on IP‑range repositories. Crawl patterns are sequential by default, with no JavaScript execution or form submission. Zibber‑v does not mimic human browser behaviour; it sends bare HTTP requests with standard headers. The official GitHub repository for Zibber’s client library (github.com/zibber/zibber‑client) states the crawler obeys the Robots Exclusion Standard and supports the Crawl‑Delay directive.

📋 robots.txt Compliance

Zibber’s documentation explicitly states that Zibber‑v honours Disallow directives found in robots.txt files, as verified by multiple independent webmaster reports. The bot also respects the Crawl‑Delay directive, reducing request frequency when specified. However, if a site’s robots.txt is unavailable or returns a 5xx error, Zibber‑v will not crawl that site until the file is properly served again, a behaviour noted in Zibber’s support articles.

🔍 Detection Indicators

The primary User‑Agent string is “Mozilla/5.0 (compatible; Zibber‑v/1.0; +https://zibber.com/bot)”. A secondary string “Zibber‑v/1.0” is used when the bot operates without a browser emulation layer. Additional identifying headers include a custom X‑Zibber‑ID header containing a unique customer token. The bot’s behaviour is characterised by rapid, sequential requests from a small set of IPs, often with Accept: text/html,application/xhtml+xml and no Accept‑Language header, which makes it distinguishable from standard browsers.

📊 Data Usage

Data collected by Zibber‑v is used exclusively for customer‑facing products: website change alerts, price monitoring dashboards, and SEO analytics. Zibber does not resell raw data or use it for AI model training, as stated in their privacy policy. The bot extracts structured content like product titles, prices, and meta descriptions, then feeds this into the customer’s chosen Zibber dashboard or API endpoint.

⚙️ Rate Limiting Policy

Because Zibber‑v can be configured to crawl at high frequency when customers choose not to set a delay, web applications are advised to rate‑limit it to prevent server overload. Threshold‑based blocking (e.g., 50 requests in 10 seconds from a single IP) is recommended as a protective measure without permanently blacklisting the bot, as Zibber provides official contact channels for rate‑limit adjustments.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.