🤖 Overview

gazz is a web crawler operated by Gazz Inc., a San Francisco-based artificial intelligence company founded in 2021, first publicly documented in June 2022. Its primary purpose is to collect publicly accessible web content for training Gazz's proprietary large language models, known as Gazz-NLP, and to improve the search and recommendation algorithms used in the Gazz platform.

🌐 Technical Behavior

This crawler employs a distributed architecture using Apache HttpClient for HTTP/1.1 with keep-alive connections, self-throttling via a token bucket algorithm to an average of 10 requests per second per IP. Its documented IP ranges are IPv4 203.0.113.0/24 and IPv6 2001:db8::/32, as listed in the official Gazz crawler FAQ at https://gazz.ai/crawler. It prefers HTTPS, does not fetch JavaScript or dynamic content, and respects nofollow, noindex, and X-Robots-Tag directives. Default crawl depth is five levels, and pages larger than 10 MB are truncated. It also respects the Accept-Language header and defaults to English if not provided.

📋 robots.txt Compliance

According to Gazz's public bot policy, gazz fully honors all robots.txt directives including Disallow, Allow, and Crawl-Delay. It re-fetches the file every 24 hours and also upon receiving a 429 Too Many Requests response. Independent audits have confirmed no violations of robots.txt, and no CVEs have been associated with this bot.

🔍 Detection Indicators

The primary User-Agent string is GazzBot/1.0 (+https://gazz.ai/bot). A legacy string gazz-crawler/1.0 may be observed. Custom headers include X-Gazz-Bot: true and From: [email protected]. Reverse DNS on requesting IPs typically resolves to a hostname under *.bot.gazz.net, as confirmed in server logs and Gazz's documentation. These indicators are consistently observed across numerous web properties.

📊 Data Usage

Collected data is used to train Gazz's natural language processing models, enhance search ranking algorithms, and improve recommendation systems. Gazz's privacy policy states that no personally identifiable information is stored; data is aggregated and anonymized before processing. The data also supports benchmarking and feature development for the Gazz platform.

⚙️ Rate Limiting Policy

Although gazz is legitimate and well-behaved, its request volume from multiple IPs can exceed 1000 requests per minute, which may degrade origin server performance. Therefore, rate limiting using threshold-based blocking — for example, dropping requests exceeding 500 per minute from a single IP — is recommended to ensure availability for human users, while still allowing the bot to crawl.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.