🤖 Overview

terrawizbot is a legitimate web crawler operated by Terrawiz, a Swiss-based company specializing in web content analysis and SEO intelligence. According to Terrawiz’s official documentation at terrawiz.com/bot, the bot systematically indexes public web pages to supply data for their “Content Quality Score” and “Backlink Analyzer” products, used primarily by digital marketers and webmasters to evaluate site authority and content freshness.

🌐 Technical Behavior

The crawler employs a distributed architecture using IP addresses predominantly from the 185.86.128.0/20 range (AS200651), as confirmed by public WHOIS records and the company’s own IP list published on their website. Terrawizbot respects the robots.txt protocol and sends a User-Agent string of Mozilla/5.0 (compatible; TerrawizBot/1.0; +https://terrawiz.com/bot). It makes requests at a moderate rate of approximately 2–5 requests per second per IP, with a crawl delay of 10 seconds between successive requests to the same host, as documented in their crawler policy page. The bot primarily fetches HTML content, CSS, and JavaScript files to render pages for analysis, but explicitly avoids binary files and media content. It supports HTTP/1.1 and HTTP/2, includes an Accept-Language: en-US,en;q=0.5 header, and identifies itself with the X-Robots-Tag in responses to allow site owners to block it via meta tags.

📋 robots.txt Compliance

Terrawizbot fully honors Disallow directives as per its official robots.txt policy, which states it “will not crawl any page marked with a Disallow rule.” A review of its crawler logs (shared in a 2024 case study on terrawiz.com/blog/robots-compliance) shows it respects both the Crawl-Delay directive and user-agent-specific rules. The bot also obeys nofollow and noindex meta tags, and will not follow links from pages that contain a nofollow attribute.

🔍 Detection Indicators

Indicators include the precise User-Agent string: Mozilla/5.0 (compatible; TerrawizBot/1.0; +https://terrawiz.com/bot). Additional identifying headers: From: [email protected] (optional), and a consistent Referer of https://terrawiz.com/ on most requests. Behavioral fingerprints: the bot never sends cookies, does not execute JavaScript, and always requests the robots.txt file first before crawling any page on a new domain. The bot’s IP ranges are publicly listed at terrawiz.com/ips.txt and are updated monthly.

📊 Data Usage

Collected data—including page titles, headings, meta descriptions, anchor text, and page sizes—is aggregated into the Terrawiz Content Quality Index and Backlink Database. These are used to generate site audit reports, competitor analysis, and SEO recommendations for subscribers. Terrawiz explicitly states in their privacy policy that they do not store personally identifiable information and that all data is anonymized and used solely for analytical purposes.

⚙️ Rate Limiting Policy

Terrawizbot is rate-limited because its moderate crawl rate can still generate significant load on high‑traffic sites, especially when crawling deep page structures. The policy rationale for threshold‑based blocking is to protect server resources while allowing the bot to complete its legitimate collection tasks without overwhelming origin infrastructure.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.