🤖 Overview

Norbert the Spider is a web crawler operated by Searchmetrics GmbH, a Berlin-based SEO and content marketing platform. First publicly documented around 2012, its purpose is to crawl websites to collect data for Searchmetrics’ SEO analytics suite, including keyword rankings, backlink profiles, and on-page optimization metrics. The bot feeds data into Searchmetrics’ SaaS products such as the Content Experience Platform and Rank Tracker.

🌐 Technical Behavior

Norbert the Spider fetches pages via HTTP/1.1 and supports both HTTP and HTTPS protocols. It honors a Crawl-Delay directive in robots.txt or defaults to approximately one request per second to remain polite. Crawl patterns focus on discovering internal links, meta tags, structured data (JSON-LD, Microformat), and visible text content. The bot’s IP addresses originate from Searchmetrics-owned ranges, typically in Germany (e.g., 95.130.40.0/21 and 87.230.48.0/20). It does not execute JavaScript, so only static HTML is indexed. Request frequency is moderate and adaptive; for large sites without explicit rate limits, it may increase gradually.

📋 robots.txt Compliance

Searchmetrics’ official documentation and third-party crawler lists (e.g., robotstxt.org) confirm that Norbert the Spider fully respects Disallow, Crawl-Delay, and Allow directives. There are no known advisories or incidents of the bot ignoring robots.txt rules. It is consistently described as a “polite crawler” by the SEO community.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Norbert the Spider; +http://www.searchmetrics.com/en/norbert/). Additional variants may include “Norbert” in the comment field. The bot sets a From header with the email [email protected]. Reverse DNS lookups reveal hostnames under searchmetrics.com or crawl.searchmetrics.com. The bot does not mask its identity and is easily identifiable.

📊 Data Usage

Collected data is used exclusively within Searchmetrics’ proprietary SEO and content analysis platform. This includes tracking keyword rankings, analyzing backlink profiles, auditing site structure, and benchmarking competitors. The data is aggregated and anonymized for client dashboards; it is not used for public AI training or sold to third parties. Searchmetrics states the data informs their “SEO visibility score” and “Content Experience” metrics.

⚙️ Rate Limiting Policy

Norbert the Spider is rate-limited because even a polite crawler can consume significant server resources on high-traffic or large sites. A threshold-based blocking policy (e.g., limiting requests to 1 per second per IP) is recommended to avoid excessive load while still allowing the bot to collect necessary SEO data. This ensures fair resource allocation for all legitimate crawlers and human visitors.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.