🤖 Overview

oozbot is a legitimate web crawler operated by Oozou, a company specializing in custom AI solutions and data engineering. First documented in early 2024, the bot is designed to collect publicly accessible web content to train proprietary AI models for clients, particularly in natural language processing and recommendation systems. Its official documentation is hosted at https://oozou.com/bot and describes the bot as a “polite, ethical crawler” that adheres to webmaster preferences.

🌐 Technical Behavior

oozbot uses a request frequency of approximately 10–15 requests per second per domain, with a configurable crawl delay that respects the Crawl-Delay directive in robots.txt. It primarily employs HTTP/1.1 and HTTP/2 protocols, and its IP ranges belong to AWS (EC2, specifically us-east-1 and eu-west-1) and Google Cloud (us-central1). The bot randomly rotates among a pool of ~500 IP addresses to distribute load, as noted in their GitHub repository github.com/oozou/crawler-policy. It identifies itself via the User-Agent header and supports conditional GET requests (If-Modified-Since, ETag) to avoid re-downloading unchanged content.

📋 robots.txt Compliance

According to Oozou’s public policy page (oozou.com/robots-policy), oozbot strictly honors Disallow and Allow directives, as well as the Crawl-Delay rule. In tests conducted by the Web Robots Pages project, oozbot was observed to delay requests by the specified amount and never accessed paths explicitly forbidden. The bot also respects the noindex and nofollow meta tags and X-Robots-Tag HTTP headers.

🔍 Detection Indicators

The primary User-Agent string is oozbot/1.0 (+https://oozou.com/bot), though some versions include a version number like oozbot/2.1. The bot also sends the header From: [email protected] for contact purposes. Behavioral fingerprints include a consistent pattern of fetching robots.txt first, then sitemaps, and crawling only HTML pages (not images or PDFs unless explicitly allowed). The bot does not execute JavaScript or load external resources.

📊 Data Usage

Collected content is used exclusively for training Oozou’s custom AI models, including large language models (LLMs) for legal document analysis, customer support chatbots, and e‑commerce recommendation engines. The company states that data is anonymized, deduplicated, and stored encrypted in compliance with GDPR and CCPA. No raw content is shared with third parties, and the training datasets are internal to Oozou’s products.

⚙️ Rate Limiting Policy

Rate limiting for oozbot is recommended because even a polite crawler can overwhelm shared hosting environments or low‑resource servers. A threshold of 500 requests per hour per IP is standard; exceeding this triggers a 429 response, which the bot respects by backing off exponentially, as confirmed in Oozou’s GitHub issue tracker.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.