🤖 Overview

SearchmetricsBot is a legitimate web crawler operated by Searchmetrics GmbH, a Berlin‑based SEO and content analytics company founded in 2005. Its primary purpose is to systematically collect publicly accessible web page data—including content, metadata, and link structures—to feed into the Searchmetrics Suite, a platform used by marketers and SEO professionals for keyword research, ranking analysis, backlink audits, and competitive benchmarking. The bot is explicitly described on Searchmetrics’ official documentation as a “non‑malicious crawler” dedicated to powering its commercial SEO tools.

🌐 Technical Behavior

SearchmetricsBot employs a link‑following crawl pattern that traverses pages starting from seed URLs or sitemaps, respecting standard HTTP conventions (User‑Agent, Accept‑Encoding, etc.). Requests are sent at a moderate rate—typically one request every 2–5 seconds—but can vary based on server response times and the site’s declared Crawl‑Delay directive. The bot uses IPv4 addresses drawn from Searchmetrics’ own autonomous system (ASN), which are publicly listed on their website and frequently updated. All requests are made over HTTP/1.1 and rarely, if ever, over HTTPS unless redirected. The crawler does not execute JavaScript, so it only captures static HTML content and links visible in the raw source.

📋 robots.txt Compliance

Searchmetrics officially states that their bot fully honors robots.txt directives, including Disallow rules and the Crawl‑Delay instruction. Evidence of this compliance is provided in Searchmetrics’ own “Crawler Overview” page, which advises website owners to use the standard robots.txt file to block specific paths. In practice, the bot fetches /robots.txt before each crawling session and obeys its rules without exception. There are no known instances of the bot ignoring disallow rules, and multiple third‑party tests (e.g., BotCheck.me) confirm its respectful behavior.

🔍 Detection Indicators

The primary User‑Agent string is SearchmetricsBot/1.0 (or simply SearchmetricsBot), and it may include a version suffix such as 1.0.0. A secondary string searchmetricsbot (lowercase) has also been reported in older log files. The bot’s IPs belong to the 195.201.0.0/16 range (Searchmetrics’ registered subnet). Behavioral fingerprints include a consistent pattern of requesting /robots.txt, then sequentially crawling pages with a default delay, and using a Accept‑Encoding: gzip, deflate header. No custom HTTP headers beyond standard ones are added.

📊 Data Usage

Data collected by SearchmetricsBot is ingested into the Searchmetrics Suite to provide real‑time SEO metrics such as keyword rankings, domain authority scores, backlink profiles, and content gap analyses. Unlike AI‑training crawlers, this data is not used to train large language models but rather to generate on‑demand reports and dashboards for paying subscribers. The company also uses aggregated, anonymised data for industry trend reports and benchmark studies, but individual page content is never republished.

⚙️ Rate Limiting Policy

SearchmetricsBot is rate‑limited because even moderate‑frequency crawlers can cause performance degradation on poorly optimized servers or during peak traffic. The policy recommends threshold‑based blocking (e.g., limiting requests to a certain number per minute per IP) to ensure server stability while still allowing the bot to collect the data necessary for its SEO analytics product. This aligns with industry best practices for non‑malicious automated agents.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.