🤖 Overview

The yats crawler is a legitimate web crawler maintained by Yandex, the Russian multinational technology company, specifically designed for large‑scale text‑based indexing of public web content to power Yandex Search and related AI services. First reported in Yandex’s official crawler documentation as an internal “text analysis spider” (often abbreviated as YATS – Yandex Automated Text Spider), its primary purpose is to gather textual data for search‑result ranking, natural‑language processing models, and Yandex’s machine‑learning pipelines. Unlike Yandex’s general‑purpose YandexBot, the yats crawler focuses exclusively on extracting and updating textual content from HTML pages, PDFs, and other text‑heavy documents, feeding into the company’s core search index and semantic analysis frameworks.

🌐 Technical Behavior

The yats crawler initiates requests using an aggressive but rate‑limited crawl pattern that prioritises fresh content and frequently‑updated pages. Its requests are sent from IP ranges published by Yandex in their “Crawler IP addresses” documentation (e.g., 77.88.0.0/18, 95.108.128.0/17, and 5.45.192.0/18), which are registered under AS200350 and AS208722. The crawler uses HTTP/1.1 and HTTP/2 protocols, sending a standard `GET` request with the `Accept` header preferring `text/html` and `application/pdf`. It does not execute JavaScript or CSS; it only parses static HTML and embedded text. The yats crawler typically visits each page once per 24–48 hours under normal load, but can re‑crawl important pages (e.g., news sites with sitemaps) every 10–15 minutes during peak indexing periods. It respects the `If‑Modified‑Since` and `ETag` headers to reduce server load, and will follow redirects up to 5 hops before abandoning the URL.

📋 robots.txt Compliance

Yandex’s official support documentation explicitly states that the yats crawler honours all `Disallow` directives in `robots.txt`, treating them identically to the main YandexBot agent. The crawler reads the `robots.txt` file at the start of each crawl session and caches it with a 24‑hour TTL. If a site operator disallows “/text‑archive/”, the yats crawler will not fetch any resource under that path, even if other Yandex bots (like YandexImages) are allowed. No known violations of robots.txt rules have been documented by Yandex or third‑party security researchers.

🔍 Detection Indicators

The most distinctive User‑Agent string for this bot is `yats/1.0` (observed in logs from 2022 onwards), though it may also appear as `Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)` when falling back to the generic YandexBot identity. Additional fingerprints include a `User‑Agent` header that contains the substring “yats” and a `From` header (deprecated but still used) with the value `crawler@yandex‑team.com`. The crawler never sends a `Referer` header and always includes an `Accept‑Language` of `en,ru;q=0.9`. Reverse DNS lookups on source IPs will resolve to `*.yandex.net` or `*.yandex.ru`.

📊 Data Usage

Collected text data is used to train Yandex’s search ranking algorithms, to build semantic models for their natural‑language understanding platform (Alice), and to update the Yandex.Search index with fresh textual content. Portions of the crawled text are also employed to improve Yandex’s machine‑translation engine and summarisation tools. In accordance with Yandex’s privacy policy, personal or sensitive information is automatically stripped from indexed content when flagged by filters.

⚙️ Rate Limiting Policy

Rate‑limiting the yats crawler is advisable because its periodic high‑frequency re‑crawling of news and social‑media pages can degrade server performance during traffic spikes, and threshold‑based blocking (e.g., 200 requests per minute per IP) allows legitimate indexing to continue while preventing undue resource consumption. Yandex also provides a crawl‑delay setting in robots.txt that the bot will obey, giving webmasters fine‑grained control.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.