🤖 Overview

watzbot is a web crawler operated by Watz, an AI-powered search engine and knowledge graph platform launched in 2023 by Watz Inc. (based in San Francisco). Its primary purpose is to index publicly accessible web content to fuel a conversational search interface and a structured knowledge base, rather than traditional keyword-based search results. The bot collects text, metadata, and structured data from web pages to train Watz's proprietary language models and improve its answer generation capabilities, as documented on the official Watz website (watz.co).

🌐 Technical Behavior

watzbot performs HTTP GET requests with a configurable crawl delay of 1 second by default, but can be adjusted via the Crawl-Delay directive in robots.txt. It respects the Host header and issues requests from IP ranges primarily in the 45.33.0.0/16 block (AS 63949, Linode), with additional addresses in the 198.58.0.0/16 and 96.126.0.0/16 ranges. The bot uses HTTP/1.1 and HTTP/2 protocols and fetches pages with a maximum size of 10 MB. It follows redirects and handles gzip, deflate, and brotli content encodings. The crawler distributes requests across multiple threads to maintain throughput while obeying Crawl-Delay and Disallow rules. It does not fetch resources like images, CSS, or JavaScript, focusing solely on HTML and XML content. The bot respects the noindex meta tag and X-Robots-Tag HTTP headers, and it includes a descriptive User-Agent string to facilitate easy identification and filtering.

📋 robots.txt Compliance

Based on official documentation at watz.co/robots, watzbot fully supports the Robots Exclusion Protocol, including User-agent, Disallow, Allow, and Crawl-Delay directives. It does not ignore any standard rules and will cease crawling any URL path listed under Disallow. The bot also respects Sitemap directives for efficient crawl prioritization. There have been no reported instances of watzbot disregarding robots.txt rules, and its compliance is explicitly verified by the company's transparency reports on their developer portal (watz.co/dev/crawler).

🔍 Detection Indicators

The primary User-Agent string is watzbot/1.0 with an additional comment (+https://watz.co/bot). A secondary User-Agent string WatzBot/1.0 (capitalized) is also used in some early deployments. The bot includes a From HTTP header containing the email address [email protected] for contact. Behavioral fingerprints include a consistent request pattern with a default crawl delay of 1 second, a narrow set of request headers (e.g., Accept: text/html,application/xhtml+xml), and no support for cookies or JavaScript. The bot's IP geolocation generally shows US West Coast origin (Linode datacenters in Fremont, CA).

📊 Data Usage

Collected data is used to build Watz's conversational search engine, which provides direct answers to user queries using real-time indexed content. The data also trains Watz's proprietary language models for improved natural language understanding and knowledge graph population. Watz explicitly states on its privacy policy (watz.co/privacy) that it does not sell user data or use the crawled content for advertising purposes. The platform deduplicates and normalizes content to reduce storage overhead and improve search accuracy.

⚙️ Rate Limiting Policy

watzbot is rate-limited by default to prevent overtaxing servers—its Crawl-Delay can be set to any positive integer, and the bot will obey it. For sites that observe aggressive crawling despite the delay, webmasters can impose a threshold-based block (e.g., 100 requests per minute) as watzbot's concurrency may still cause load on shared hosting environments. The company encourages proactive rate limiting rather than blocking, as the bot is legitimate and compliant.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.