🤖 Overview

WebLinker is an automated web crawler operated by Yandex (via its search and indexing division) as part of the YandexBot family, first documented in Yandex’s official crawler list. Its primary purpose is to discover and verify outbound hyperlinks from crawled pages, updating the Yandex web graph to improve link-based ranking algorithms and feed the Yandex search index with accurate link relationships.

🌐 Technical Behavior

WebLinker performs a focused, link‑centric crawl distinct from YandexBot’s full‑page download; it typically accesses only the HTML response to extract anchor tags, ignoring embedded resources such as images, CSS, or JavaScript. Crawl frequency is moderate—typically one request per several seconds per host—as documented in Yandex’s robot.txt guidelines. IP ranges are drawn from Yandex’s public ASN AS20821 and AS13238, with reverse DNS records resolving to yandex.net. The bot uses HTTP/1.1 and HTTP/2 protocols, sends an Accept-Language: ru, en header, and may issue conditional GET requests (If‑Modified‑Since) to respect server caching.

📋 robots.txt Compliance

Yandex officially states that WebLinker respects the Disallow directives in robots.txt, as confirmed in the Yandex help article “Crawler user agents” (yandex.com/support/robot). It also obeys the Crawl-Delay directive. However, because it is a separate user agent, site operators must explicitly add User-agent: YandexWebLinker rules if they wish to block it independently from YandexBot.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; YandexWebLinker/1.0; +https://yandex.com/bots). Behavioral fingerprints include a low request rate per IP, absence of resource sub‑requests, and a consistent From header of [email protected]. The bot also sends a X-Robots-Tag parsing preference and does not accept cookies.

📊 Data Usage

Collected link data feeds Yandex’s MatrixNet ranking algorithm, influencing result order by evaluating link quality and freshness. Additionally, link metadata is used to populate Yandex’s Site Map service and to detect broken links for site owners via Yandex Webmaster Tools.

⚙️ Rate Limiting Policy

Because WebLinker is a high‑volume crawler that can inadvertently impact server performance—especially on sites with thousands of pages—rate‑limiting at 5–10 requests per second per IP is recommended. This threshold prevents resource exhaustion while still allowing legitimate link discovery, aligning with Yandex’s own guidance to use the Crawl-Delay directive.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.