🤖 Overview

Mail.RU_Bot is a web crawler operated by the Russian internet company Mail.Ru Group (now part of VK), primarily used to index web pages for the Mail.ru search engine (go.mail.ru). According to official documentation available at http://go.mail.ru/help/robots, the bot also supports other Mail.ru services such as content categorization and ranking data for the portal’s news aggregation. First observed in the early 2000s, it remains an active, legitimate crawler that respects standard web protocols.

🌐 Technical Behavior

Mail.RU_Bot sends HTTP/1.1 GET requests with standard headers (User-Agent, Accept, Accept-Encoding). Its crawl pattern is sequential, often starting from a root URL and following internal links up to a configurable depth (default 3). The bot uses a rotating pool of IP addresses from Mail.Ru’s owned ASN (AS47764) and sometimes from cloud providers; reported ranges include 94.100.176.0/20 and 185.5.0.0/16. Historical analysis shows request intervals typically between 1 and 5 seconds, but it respects the Crawl-delay directive if set. It does not parse JavaScript or execute dynamic content, focusing only on static HTML and linked resources (CSS, images) for indexing.

📋 robots.txt Compliance

Mail.Ru explicitly states on its help/robots page that Mail.RU_Bot fully obeys the Robots Exclusion Protocol, including Disallow, Allow, and Crawl-delay directives. Independent tests by webmasters (e.g., posts on WebmasterWorld, 2019) confirm the bot reduces its request rate when a Crawl-delay: 2 is specified. There are no verified reports of it ignoring Disallow rules, though some sites have observed it occasionally re-crawling blocked pages after long intervals, likely due to cached directive data.

🔍 Detection Indicators

The primary User-Agent strings are "Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)" and the older "Mail.RU_Bot/1.0". Additional identifying headers include a From: header (rare) or X-Forwarded-For when behind a proxy. IP-based detection can use reverse DNS lookups ending in .mail.ru or .corp.mail.ru. Behavioral fingerprints include sequential crawling without concurrent requests and a 5-10 second delay after receiving a 429 status (rate limiting response).

📊 Data Usage

Collected data feeds the Mail.ru search engine index, improving its ranking and freshness for Russian-language and global web pages. Additionally, indexed content is used for internal analytics (trending topics, content duplication detection) and possibly for advertising targeting — though Mail.ru’s privacy policy states it does not store personal identifiable content. The bot does not train large language models; its output is solely for search and related services.

⚙️ Rate Limiting Policy

Mail.RU_Bot is rate-limited because it can send up to 2–3 requests per second without explicit delay, which may overwhelm smaller servers. Administrators are advised to set a Crawl-delay: 2 in robots.txt to reduce load, and threshold-based blocking (e.g., 429 status after 10 req/sec) is a recommended safety measure to prevent unintended resource exhaustion while still allowing legitimate indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.