🤖 Overview

SISTRIX is a web crawler operated by the SISTRIX GmbH based in Cologne, Germany, primarily powering the SISTRIX SEO tool suite (sistrix.com). According to SISTRIX’s official documentation, the bot collects publicly accessible web data to provide search engine optimization analytics, including keyword rankings, backlink profiles, on-page audits, and SERP feature monitoring. The crawler is a legitimate commercial agent serving digital marketers and website owners, not a malicious actor.

🌐 Technical Behavior

The SISTRIX crawler identifies itself via the User-Agent string "SISTRIX" or "SISTRIX Crawler" (see official SISTRIX support page at https://www.sistrix.com/robots.txt). It respects a default crawl delay of 10 seconds between requests (configurable via robots.txt Crawl-Delay directive). The bot uses IPv4 addresses predominantly from German data centers, with IP ranges documented in SISTRIX’s published IP list (e.g., 144.76.x.x, 5.9.x.x ranges of Hetzner). It supports both HTTP/1.1 and HTTPS, and sends a Referer header pointing to sistrix.com. The crawler typically fetches HTML pages, CSS, and JavaScript (to render pages for SEO analysis), but does not download images or large binary files. SISTRIX states that the bot performs a full site crawl for each domain it analyzes, with a frequency that depends on the customer’s subscription plan.

📋 robots.txt Compliance

SISTRIX formally honors the robots.txt Disallow directives as stated in their official FAQ (https://www.sistrix.com/faq/crawler/). They also support the Crawl-Delay directive to further throttle requests. If a site blocks the SISTRIX User-Agent via robots.txt, the crawler will cease all activity on that domain. However, SISTRIX does warn that cached data from previous crawls may remain in their database until the next scheduled recrawl.

🔍 Detection Indicators

Key detection indicators include the User-Agent strings: "SISTRIX", "Mozilla/5.0 (compatible; SISTRIX; +https://www.sistrix.com)", and for JavaScript rendering: "SISTRIX; JS". The bot also sends a Via header containing "SISTRIX HTTP Proxy". A behavioral fingerprint is the crawl pattern: it will request a robots.txt file first, then systematically fetch page URLs with a steady 10-second interval (unless a different crawl delay is set). SISTRIX does not simulate human mouse movements or click events.

📊 Data Usage

Collected data is exclusively used for the SISTRIX SEO platform, which provides keyword tracking, backlink analysis, site audits, and competitive research. According to SISTRIX’s privacy policy (https://www.sistrix.com/privacy/), the data is stored pseudonymously and processed in Germany. The bot does not store personal or sensitive user data. Its purpose is to aggregate publicly available web information to generate SEO metrics for paying customers.

⚙️ Rate Limiting Policy

Rate-limiting SISTRIX is justified because its aggressive crawl schedule—crawling entire site structures with minimal delays—can consume significant server resources, especially on small to medium websites. A rate limit of 10–20 requests per minute with a burst cap of 2 concurrent connections is recommended to balance data collection needs with server stability, per industry best practices for SEO crawlers. SISTRIX supports the Crawl-Delay directive, making it easy for webmasters to enforce throttling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.