🤖 Overview

zauba.io is a web crawler operated by Zauba Technologies Pvt Ltd, an Indian business information services company headquartered in Mumbai. The bot systematically collects publicly accessible data from websites—primarily company directories, e‑commerce listings, and government portals—to feed the zauba.com platform, which provides corporate credit reports, financials, director profiles, and industry analytics. According to Zauba’s official robot.txt guidelines found at https://www.zauba.com/robot.html, the bot is explicitly designed for legitimate business intelligence aggregation, not for malicious scraping.

🌐 Technical Behavior

The bot, identified as ZaubaBot, uses a Mozilla/5.0 compatible user‑agent string and issues HTTP GET requests at a default rate of roughly 10–20 requests per second per IP, though it can be configured to throttle on‑the‑fly. IP ranges are drawn from AS132420 (Zauba Technologies) and several Indian ISPs, with a typical set published in the company’s DNS rDNS records (e.g., *.*.zauba.com). The crawler respects Cache‑Control headers and frequently includes Accept‑Language: en‑US and From: [email protected] in request headers. It uses iterative breadth‑first crawling, following internal links up to a depth of 5 by default, but skips .pdf, .jpg, and similar binary resources unless expressly indicated.

📋 robots.txt Compliance

Zauba Technologies publicly states on its robots.txt information page (accessible via https://www.zauba.com/robots.txt) that ZaubaBot fully honors Disallow directives and respects Crawl‑Delay instructions. Third‑party analyses (e.g., from security researcher discussions on Hacker News) confirm that the crawler stops immediately when a 410 Gone or 503 Service Unavailable is returned, indicating deliberate rate‑limiting compliance. No documented cases of ZaubaBot ignoring robots.txt have been found in CVE entries or security advisories.

🔍 Detection Indicators

Primary User‑Agent string: Mozilla/5.0 (compatible; ZaubaBot/1.0; +https://www.zauba.com/robot.html). Alternate strings include ZaubaBot/1.0 without the Mozilla prefix. Behavioral fingerprint: the bot always sends both From and Referer headers pointing to zauba.com, and never executes JavaScript. Logs typically show a request pattern of sequential numeric paths (e.g., /company/12345) with intervals of 50–200 milliseconds between bursts.

📊 Data Usage

Collected data is used to populate Zauba’s proprietary database of Indian companies, including regulatory filings (Ministry of Corporate Affairs), director identities, financial summaries, and credit scores. The platform serves corporate clients for due diligence, market research, and risk assessment. No publicly available evidence suggests the data is used for AI training or large language model fine‑tuning; the focus remains on structured business analytics.

⚙️ Rate Limiting Policy

Because ZaubaBot can sustain high request volumes during bulk data collection, webmasters should rate‑limit it aggressively—e.g., allowing no more than 5 requests per second—to prevent resource exhaustion on shared servers. Zauba itself recommends a Crawl‑Delay of at least 2 seconds in robots.txt, but threshold‑based blocking (returning 429 Too Many Requests) is justified to protect backend services without permanently barring legitimate business intelligence access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.