🤖 Overview

Iskanie is a web crawler operated by OOO "Iskanie", the company behind the Russian search engine Iskanie.ru (also known as "Искание"). First launched in 2014, its primary purpose is to index publicly accessible web content to power the Iskanie search engine, which focuses on Russian-language results and regional relevance. The bot is considered a legitimate, non-malicious agent that follows standard crawling protocols, though it may exhibit aggressive indexing behavior on sites with high link density.

🌐 Technical Behavior

Iskanie crawls using HTTP/1.1 with support for both HTTP and HTTPS protocols, typically sending requests at a rate of 5–10 requests per second per host, though this can spike during initial discovery phases. It identifies links via HTML anchor tags, sitemaps, and robots.txt references. The bot respects noindex meta tags but does not always obey nofollow on outbound links. IP addresses are allocated from Russian autonomous systems, primarily from AS200350 (Iskanie LLC) and AS197068 (subsidiary ranges). Crawls are distributed across multiple IPs to reduce per-IP load on target servers. The bot uses HTTP/1.1 Keep-Alive to reduce connection overhead and parses JavaScript-rendered content only when explicitly enabled via a X-Robots-Tag: nojs header override.

📋 robots.txt Compliance

According to public documentation from the Iskanie webmaster portal (iskanie.ru/webmaster), the bot fully honors Disallow, Allow, and Crawl-delay directives in robots.txt. However, it does not support the Disallow: / wildcard for entire-site blocking if a Crawl-delay is also present; the delay is applied first. There have been community reports (e.g., on search-engine-land.ru) that the bot occasionally ignores Disallow for subdirectories during high-priority recrawl cycles, though Iskanie officially denies this.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Iskanie/1.0; +https://iskanie.ru/bot), though some crawls use Iskanie/1.0 without the Mozilla prefix. Additional identifying headers include From: [email protected] and a X-Forwarded-For containing a typical Russian IP. Behavioral fingerprints include a consistent request interval of 2–5 seconds per resource and a lack of Accept-Language header in many requests.

📊 Data Usage

Collected page content is stored and processed exclusively for the Iskanie search index, which serves Russian-language search queries across Baidu, Yandex, and Google-like verticals. The data is also used to generate page snippet previews and to compute relevance scores based on tf-idf weighting. Iskanie does not use scraped content for AI model training or commercial analytics; it is solely employed to improve organic search rankings within its own ecosystem, as stated in its privacy policy (iskanie.ru/privacy).

⚙️ Rate Limiting Policy

Rate limiting is recommended because Iskanie may not respect Crawl-delay under heavy server load, and its bursty indexing patterns can overwhelm lower-capacity sites. A threshold-based blocking approach (e.g., 20 requests per minute per IP) prevents degraded performance while still allowing legitimate indexing to proceed uninterrupted, aligning with standard SEO best practices for search engine bots.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.