🤖 Overview

Metager2 is a web crawler operated by MetaGer GmbH, the German company behind the privacy-focused search engine MetaGer (metager.de). Unlike many commercial search bots, Metager2 was developed specifically to support MetaGer's open-source, privacy-respecting index, which sources results from its own crawl rather than relying solely on Bing or Google APIs. The bot's primary purpose is to collect publicly accessible web pages to build and refresh MetaGer's search index, ensuring users receive independent, unbiased search results. Official documentation is available at https://metager.de/bot and the project's code is hosted on the MetaGer GitHub repository (https://github.com/MetaGer).

🌐 Technical Behavior

Metager2 follows standard HTTP/1.1 and HTTP/2 protocols, sending requests with a default crawl delay configured to avoid overwhelming servers. Based on published logs, the bot typically issues requests at a rate of one request every few seconds per domain, though the exact frequency may vary depending on server response times and robots.txt directives. The crawler uses IPv4 addresses from German IP ranges (registered under AS for MetaGer or its hosting provider), and it may also employ IPv6 addresses. It respects If-Modified-Since and ETag headers to reduce redundant downloads, and it only crawls pages linked from the public web or submitted via MetaGer's URL submission form. The bot does not execute JavaScript or follow redirects that point to non-public resources.

📋 robots.txt Compliance

Metager2 is documented as fully compliant with the Robots Exclusion Standard. According to the official bot page at https://metager.de/bot, the crawler reads and respects Disallow directives in robots.txt before each request and will not access any URL path that is forbidden. It also honors Crawl-Delay directives if present, adjusting its request interval accordingly. Administrators can block the bot entirely by adding User-agent: Metager2 with a Disallow: / directive.

🔍 Detection Indicators

The primary User-Agent string reported by Metager2 is Mozilla/5.0 (compatible; Metager2/1.0; +https://metager.de/bot). Some older versions may use Metager2/1.0 without the Mozilla prefix. The bot also sends a From header containing [email protected] or a contact email. Behavioral fingerprints include a consistent request pattern with no JavaScript dependency, a low frequency of requests, and a strong preference for HTML and text content over images or binary files.

📊 Data Usage

All data collected by Metager2 is used exclusively to populate the MetaGer search index, which is then served to users via the meta search engine. The index is publicly accessible and is not used for any AI training, advertising profiling, or third-party data sales. MetaGer operates under strict German privacy laws (GDPR) and does not log IP addresses longer than necessary. The crawled content is stored temporarily for indexing and is refreshed periodically.

⚙️ Rate Limiting Policy

Although Metager2 is a legitimate, non-malicious bot, it is rate-limited on many sites because its crawl speed, while modest, can still degrade server performance if not throttled. The recommended policy is to allow the bot with a moderate rate limit (e.g., 10 requests per minute per IP) and only block it if it causes excessive load, as documented in the MetaGer bot ethics guidelines at https://metager.de/bot.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.