🤖 Overview

The webcapture bot is a legitimate web crawler operated by Yandex as part of their search infrastructure, specifically designed to capture and store visual representations of web pages for use in search result snippets and previews. This bot was first documented in Yandex’s official crawler list around 2019 and primarily feeds data into Yandex’s public search engine, enriching the user experience with thumbnail images and layout captures. According to Yandex’s support portal (https://yandex.com/support/webmaster/robot-workings/), it is distinct from the main YandexBot and operates under a separate user-agent string to allow site owners granular control via robots.txt.

🌐 Technical Behavior

The webcapture bot uses a headless browser engine to render pages fully before taking a snapshot, which means it downloads all resources including CSS, JavaScript, and images. It initiates requests from Yandex’s known IP ranges (e.g., 77.88.0.0/18 and 93.158.134.0/23, per official Yandex IP lists at https://yandex.com/support/webmaster/robot-workings/ip-addresses.html). The crawl pattern is moderate: it typically issues requests every 2–5 seconds per domain and respects the `Crawl-Delay` directive in robots.txt. Unlike text-only crawlers, this bot may re-crawl pages more frequently if the page layout changes, as its purpose is to keep visual captures up-to-date. It communicates over HTTPS only and sends a `User-Agent: YandexWebCapture/1.0` header along with a `From: [email protected]` email for abuse contact.

📋 robots.txt Compliance

Yandex explicitly states in their webmaster guidelines that the webcapture bot honors the `Disallow` and `Crawl-Delay` directives in robots.txt, provided that the user-agent line matches exactly `YandexWebCapture`. In testing by various SEO practitioners, the bot has been observed to respect `Disallow: /private/` and similar rules without fail. Site owners can also block the entire bot by using `User-agent: YandexWebCapture` followed by `Disallow: /`.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: `Mozilla/5.0 (compatible; YandexWebCapture/1.0; +http://yandex.com/bots)` or `YandexWebCapture/1.0` in older versions. A secondary fingerprint is the `Accept-Language` header set to `ru-RU,ru;q=0.9,en;q=0.8` and the absence of typical human interaction headers like `Sec-Fetch-User`. Server logs may also show a consistent request pattern with a `Referer` header set to `http://yandex.com/bots`.

📊 Data Usage

The captured visual data—full-page screenshots and layout metadata—is used exclusively to generate eye-catcher thumbnails in Yandex search result pages (SERPs). This data is not used for AI training or behavioral analysis; Yandex’s privacy policy (https://yandex.com/legal/confidential/) confirms that visual captures are ephemeral and refreshed regularly to reflect current page state. The bot also collects the page’s rendered size and aspect ratio to optimise thumbnail presentation.

⚙️ Rate Limiting Policy

While the bot is non-malicious, it can consume significant bandwidth when crawling media-heavy sites, which is why rate limiting is recommended. Implementing a threshold of 5 requests per minute per IP from the Yandex bot range ensures fair resource allocation while still allowing the bot to complete its snapshot tasks for improved search visibility.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.