🤖 Overview

GermCrawler is the proprietary web crawler operated by GermanSearch GmbH (formerly GermCrawler GmbH), a German search engine and web directory provider founded in 2018. Its primary purpose is to index public web content—including HTML pages, PDFs, images, and videos—for the GermanSearch search engine (germancsearch.com) and related vertical search products. The bot was first publicly documented in 2019 and is exclusively used for building and refreshing the search index, not for AI training or data resale. According to official documentation at germansearch.com/crawler, the bot adheres to German data protection laws (DSGVO) and respects robot exclusion standards.

🌐 Technical Behavior

GermCrawler performs both broad and targeted crawls using a distributed architecture. It typically requests pages at a moderate rate of 1–5 requests per second per IP, with bursts up to 10 req/s during initial discovery phases. The crawler uses HTTP/1.1 and HTTP/2 protocols, sends a valid From header identifying the operator email ([email protected]), and supports If-Modified-Since and ETag for bandwidth efficiency. IP ranges are publicly listed in an SPF-style record at crawler.germansearch.com/ip-ranges.txt and primarily originate from German datacenters (AS24961, AS197540). The bot may appear from multiple distinct IPv4 addresses within the 88.198.0.0/16 and 5.104.64.0/19 ranges, but it always identifies itself via the User-Agent string. It does not run JavaScript and only fetches static resources (CSS, images) needed for index metadata.

📋 robots.txt Compliance

GermCrawler fully honors robots.txt directives as documented in its official guidelines. It checks the file before each crawl session and respects Disallow, Crawl-Delay, and Allow rules with a minimum delay of 5 seconds when specified. The operator explicitly states that any Disallow rule is enforced without exception, and webmasters can also block via X-Robots-Tag HTTP header. There is no evidence of the bot ignoring exclusion signals; third-party analyses (e.g., BotScout reports) confirm compliance.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; GermCrawler/2.0; +https://germansearch.com/crawler). Older versions used GermCrawler/1.0. The bot sends a User-Agent header that always contains "GermCrawler" and a From header with the operator email. It does not spoof common browser user agents. Behavioral fingerprints include a low request rate, lack of Accept-Language diversity, and consistent use of Accept: text/html,application/xhtml+xml.

📊 Data Usage

Collected data is used exclusively for building and updating the GermanSearch search index, including page titles, descriptions, headings, and text content. Metadata (e.g., last-modified dates, content type) is stored to improve search relevance and freshness. No content is shared with third parties, sold, or used for training large language models. The operator publishes a transparency report quarterly detailing crawl volume and domain distribution.

⚙️ Rate Limiting Policy

GermCrawler is rate-limited because it can aggressively re-crawl pages after a refresh cycle, sometimes hitting site limits. Recommended threshold-based blocking (e.g., >10 req/s from the same IP range over 60 seconds) is justified to protect server resources while still allowing the bot to index content. The operator recommends a delay of 5 seconds in robots.txt to keep the crawler polite.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.