🤖 Overview

The Spbot crawler is operated by Sputnik, a Russian search engine owned by the state corporation Rostelecom and launched in 2014. Its primary purpose is to index publicly accessible web pages for the Sputnik search service, which aims to provide an alternative to global search engines. Spbot collects content to populate Sputnik’s search index and is considered a standard, legitimate search engine crawler.

🌐 Technical Behavior

Spbot crawls websites using HTTP/1.1 and HTTP/2 protocols, sending GET requests to retrieve HTML pages and associated resources. The bot originates from IP addresses allocated to Rostelecom, specifically within the AS8342 (Rostelecom) and AS12389 (Rostelecom) autonomous systems, with ranges visible in public WHOIS data. Crawling frequency is moderate, typically between one to ten requests per second on a single domain, and the bot respects a configurable crawl delay if set in robots.txt. Spbot does not fetch JavaScript-rendered content or execute client-side code, focusing only on static HTML. The crawler identifies itself via the User-Agent string “Mozilla/5.0 (compatible; Spbot/1.0; +http://sputnik.ru/)”, as documented on Sputnik’s official webmaster page. It also sends a “Via” header and maintains a persistent connection for efficiency.

📋 robots.txt Compliance

According to Sputnik’s official webmaster guidelines (sputnik.ru/webmaster), Spbot fully honors the robots.txt standard, including Disallow directives and the Crawl-Delay directive. The bot checks the robots.txt file at the beginning of each crawl session and caches it for up to 24 hours. There are no documented cases of Spbot ignoring disallowed paths; it is considered compliant with industry standards.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: “Spbot/1.0 (compatible; +http://sputnik.ru/)” or “Mozilla/5.0 (compatible; Spbot/1.0; +http://sputnik.ru/)”. Additional behavioral fingerprints include a source IP from Rostelecom’s ASN ranges (AS8342, AS12389) and a request pattern that omits JavaScript or CSS resources. The bot does not send a custom X-Robots-Tag header. It also includes a “Referer” header pointing to sputnik.ru for tracking.

📊 Data Usage

Collected data is used exclusively for building and updating the Sputnik search index, which provides organic search results to users of the sputnik.ru search engine. The content is stored, cached, and analyzed for ranking algorithms. Sputnik does not use the crawled data for AI training or commercial resale; its sole purpose is to deliver relevant web search results to its audience.

⚙️ Rate Limiting Policy

Spbot is rate-limited because it can generate sustained request volumes (up to 10 req/s) that may overwhelm smaller websites without proper crawl delay configuration. Administrators are advised to set Crawl-Delay: 5 in robots.txt to reduce load, and threshold-based blocking is justified when the bot exceeds 20 requests per second without honoring the directive, as this indicates misconfiguration or aggressive crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.