🤖 Overview

Mail.ru is a web crawler operated by the Russian internet company Mail.Ru Group (now part of VK), primarily used to index web content for its search engine at go.mail.ru. The crawler, officially documented under the user-agent Mail.RU_Bot, collects publicly accessible pages to build the search index that powers Mail.ru’s search results and related services such as news aggregation and image search. According to Mail.ru’s official robots.txt instructions at https://go.mail.ru/help/robots, the bot is designed to be a standard search engine crawler and does not collect data for AI training or other secondary purposes beyond search indexing.

🌐 Technical Behavior

The Mail.ru crawler follows typical HTTP/1.1 and HTTPS protocols and respects robots.txt directives. It requests pages at a moderate rate, typically sending one request every few seconds from a range of IP addresses allocated to Mail.ru’s hosting infrastructure in Russia. According to publicly available IP lists (e.g., from ipinfo.io and bgp.he.net), the crawler originates from the ASN 47764 (OOO Mail.Ru) with IP ranges such as 95.163.0.0/17, 94.25.168.0/24, and 5.61.16.0/21. The crawler uses HTTP/1.1 with standard headers including Accept-Language: ru-RU and User-Agent strings. It does not appear to follow JavaScript-rendered content but will parse standard HTML and CSS inline links.

📋 robots.txt Compliance

Mail.ru’s documentation explicitly states that the bot honors Disallow directives in robots.txt and recommends webmasters add rules such as User-agent: Mail.RU_Bot followed by Disallow: /private to restrict access. Tests by various security researchers (e.g., in community forums) confirm the bot respects these directives and does not ignore them. However, like many major search crawlers, it may cache a copy of robots.txt for up to 24 hours before re-fetching.

🔍 Detection Indicators

The primary user-agent string is Mozilla/5.0 (compatible; Mail.RU_Bot; +http://go.mail.ru/help/robots). A secondary variation may include Mozilla/5.0 (compatible; MailRuBot; +http://go.mail.ru/help/). The bot sets the From header as [email protected] in some cases. Behavioral fingerprints include a consistent request frequency of 2–5 seconds between pages and a distinct reverse DNS hostname pattern like crawler-xxx.mail.ru. It does not send a custom X-Robots-Tag header or accept-language other than Russian.

📊 Data Usage

The collected data is exclusively used to build and update the Mail.ru search index for go.mail.ru. The company states that the bot does not scrape content for training machine learning models, personalized advertising, or any AI-related tasks. The indexed data is used to provide organic search results, featured snippets, and news categorization. Mail.ru also uses the data for its “Поиск” (Search) product, which serves Russian-language queries predominantly.

⚙️ Rate Limiting Policy

Rate limiting is appropriate because the Mail.ru crawler, while legitimate, can still generate high volumes of requests if left unchecked, potentially degrading server performance. A threshold of 10 requests per second from a single IP or cluster of IPs is recommended, with a 429 response after sustained bursts, to protect application stability without completely blocking the bot’s necessary indexing functions.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.