🤖 Overview

webindex is a web crawling agent operated by Yandex, the Russian internet services company, for its search engine indexing pipeline. First documented in 2010, this bot systematically retrieves publicly accessible web pages to populate Yandex Search results and ancillary services such as Yandex.Maps and Yandex.Images (source: Yandex Webmaster Help, webmaster.yandex.com/resources/crawler-role.html).

🌐 Technical Behavior

The YandexWebIndex crawler sends HTTP/1.1 and HTTP/2 requests from IP addresses registered under Yandex’s ASN (AS20864), commonly in the 213.180.204.0/24 and 93.158.134.0/24 ranges. It respects the Crawl-Delay directive in robots.txt, with a default delay of 10–30 seconds between successive requests. The bot uses conditional GET requests with If-Modified-Since and ETag headers to reduce bandwidth, and supports gzip, deflate, and Brotli encoding. During peak indexing, a single IP may issue 10–20 requests per second, but Yandex’s own rate controls throttle overall crawl speed (Yandex Webmaster technical specification). It also fetches sitemap.xml files to discover new URLs.

📋 robots.txt Compliance

Yandex explicitly states that webindex fully obeys Disallow and Allow directives in robots.txt, caching the file for up to 24 hours. Official documentation (webmaster.yandex.com/resources/robots-txt.html) confirms that violations are extremely rare and typically due to server misconfigurations rather than intentional non-compliance.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; YandexWebIndex/1.0; +http://yandex.com/bots). Additional strings include YandexBot/3.0 and YandexWebIndex/1.0. Behavioral fingerprints include sequential sitemap requests, a consistent Accept-Encoding header (gzip, deflate, br), and a preference for text/html responses. The bot sometimes includes a From header with an administrative email address.

📊 Data Usage

Collected data is primarily used to build and update Yandex’s search index, improving result relevance and ranking algorithms. Additionally, Yandex may use the content for training machine learning models related to natural language understanding and ad targeting, as described in Yandex’s privacy policy (yandex.com/legal/confidential).

⚙️ Rate Limiting Policy

Because webindex can generate high request volumes, especially during large-scale reindexing, rate limiting using per-IP thresholds (e.g., 20 requests per second) is recommended to prevent server overload while still allowing legitimate indexing. Yandex provides guidelines for webmasters to set crawl rates via robots.txt (webmaster.yandex.com/resources/crawler-role.html).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.