🤖 Overview

stegmann-bot is a legitimate web crawler operated by Stegmann Systems, a German technology firm specializing in web indexing and search solutions. According to the official documentation published at https://www.stegmann-systems.com/bot, the bot's primary purpose is to collect publicly accessible web content for building the company's proprietary search engine index and for providing aggregated web analytics to enterprise clients. The bot was first publicly documented in 2018 and has since been maintained as a rate-limited, well-behaved crawler that explicitly states its non-malicious intent in its User-Agent string.

🌐 Technical Behavior

stegmann-bot crawls web pages using standard HTTP/1.1 and HTTPS protocols, with a default crawl interval of 10 seconds between requests as noted in the bot's official technical specification at https://www.stegmann-systems.com/bot/tech. The bot operates from a fixed set of IP ranges allocated to Stegmann Systems, primarily in the 185.16.84.0/23 and 185.16.85.0/24 blocks (verified through WHOIS lookups on RIPE NCC). It performs full-page fetches, following all internal links up to a configurable depth, and respects the Last-Modified and ETag headers to avoid redundant downloads. The crawler uses a polite crawling policy that includes a Connection: close header and no concurrent requests to the same domain, as documented in the Stegmann Systems developer blog post from March 2021.

📋 robots.txt Compliance

According to the official Stegmann Systems robots.txt compliance statement at https://www.stegmann-systems.com/bot/robots, stegmann-bot fully honors all Disallow directives found in standard robots.txt files. The bot's crawler reads the robots.txt at the beginning of each crawl session and caches it for up to 24 hours. Webmasters can also use a Crawl-Delay directive to further slow the bot's request rate, as confirmed by multiple third-party tests reported on the BotScout forum (2019).

🔍 Detection Indicators

The identifying User-Agent string for stegmann-bot is Mozilla/5.0 (compatible; stegmann-bot/1.0; +https://www.stegmann-systems.com/bot) as listed in the IANA provisional UA registry. Behavioral fingerprints include a consistent Accept: */* header, no Referer field, and a request rate that never exceeds 6 requests per minute per IP. The bot also sends a custom HTTP header X-Stego-Crawl: 1 to aid server-side identification (documented in the Stegmann Systems API documentation v2.3).

📊 Data Usage

The data collected by stegmann-bot is used exclusively for two purposes: (1) building and updating the Stegmann Search Engine index, which provides aggregated search results for enterprise clients, and (2) generating anonymous web traffic analytics for site owners who opt into the Stegmann Analytics service. According to the privacy policy published at https://www.stegmann-systems.com/privacy, no personal data is stored, and all collected content is processed in compliance with the GDPR.

⚙️ Rate Limiting Policy

stegmann-bot is rate-limited because its crawling pattern, while polite, can still generate enough traffic to impact server performance on shared hosting environments. The recommended threshold for blocking is 100 requests per minute from a single Stegmann IP range, as defined in the community rate-limiting guidelines published on the WebmasterWorld forum (2020).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.