🤖 Overview

Web Sauger is a legitimate web crawler operated by the German internet company 1&1 Ionos (formerly United Internet) for the Web.de search engine and portal. First publicly documented in the early 2000s, its purpose is to index publicly accessible web content to feed into Web.de’s search results and related services. The name "Sauger" is German for "suction device," reflecting the crawler’s function of pulling data from websites.

🌐 Technical Behavior

Web Sauger performs standard HTTP GET requests to pages, following links recursively to build an index. Its crawl frequency is moderate, typically making requests at intervals of several seconds to minutes based on server response. According to Web.de's official documentation, the bot uses IP ranges from 1&1 Ionos, such as 217.72.0.0/16 and 80.1.0.0/16. It supports HTTP/1.1 and HTTPS, includes an Accept-Language header set to German, and does not execute JavaScript. The crawler only indexes static HTML content and sends a User-Agent string ending with http://web.de/.

📋 robots.txt Compliance

Web Sauger is documented to fully honor robots.txt directives. The Web.de crawler FAQ states it reads Disallow rules, and site owners can block paths using User-agent: WebSauger. Isolated reports suggest it may occasionally ignore crawl-delay directives if set too aggressively, but such behavior is not typical.

🔍 Detection Indicators

The primary User-Agent string is WebSauger/1.0 or variants like Mozilla/5.0 (compatible; WebSauger/1.0; +http://web.de/). Additional identifying headers include a From header and a Referer header set to the originating page. The bot’s IP addresses reverse-resolve to hostnames like *crawl*.web.de. Behavioral fingerprints include a default minimum interval of 5 seconds between requests.

📊 Data Usage

Collected content is used exclusively for the Web.de search engine index, serving search results for the German-language portal. Snippets and metadata may also appear in Web.de’s news aggregator and image search. According to Web.de’s privacy policy, no data is shared with third parties for AI training or advertising beyond indexing.

⚙️ Rate Limiting Policy

Web Sauger is rate-limited by its own design to a maximum of 20 requests per minute per domain. The rationale for threshold-based blocking is to prevent excessive load on smaller websites while allowing thorough indexing. Site administrators may apply additional server-level rate limiting if the bot impacts performance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.