🤖 Overview

FreshLinks is a web crawler operated by FreshIndex, a Russian-language search engine launched in 2015, designed to index newly published or updated content from news sites, blogs, and forums within minutes. Its primary purpose is to feed near-real-time search results into the FreshIndex search product, prioritizing timeliness over comprehensive coverage.

🌐 Technical Behavior

The bot uses a distributed crawling infrastructure with IP ranges originating from Russian data centers, including ASNs assigned to Yandex (AS200350) and Selectel (AS49505). Requests are made over HTTP/1.1 with a default Accept-Language: ru header to favor Russian-language pages. FreshLinks sends multiple requests per second to the same host without a built-in delay, leading to bursty traffic patterns documented in webmaster forum posts. It supports conditional GET requests via If-Modified-Since headers to reduce bandwidth usage, but this behavior is inconsistent across server responses.

📋 robots.txt Compliance

According to the official bot documentation at https://freshindex.com/crawler.html, FreshLinks claims to honor Disallow directives and Crawl-Delay settings. However, community reports on WebmasterWorld and similar forums note occasional violations, particularly with Disallow: / being ignored for subdirectories, likely due to caching bugs in the crawler’s parsing logic rather than malice.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; FreshLinks/1.0; +http://freshindex.com/bot.html). Additional fingerprints include a missing Referer header, a high request rate of 10–20 requests per second, and IP addresses in the 95.213.0.0/16 and 178.170.0.0/16 ranges. The bot requests robots.txt only once per domain and does not include From or Via headers.

📊 Data Usage

Collected data is exclusively used to populate the FreshIndex search engine, providing users with up-to-date results for time-sensitive queries. No evidence exists of AI training, resale, or third-party sharing, aligning the bot with standard search-engine crawlers like Bingbot or YandexBot.

⚙️ Rate Limiting Policy

Because FreshLinks does not self-throttle by default, web applications apply rate limiting to prevent resource exhaustion. A threshold of 15 requests per second per IP is recommended, balancing the need for timely indexing with server stability, as documented in FreshIndex’s own crawl guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.