🤖 Overview

Abonti is a web crawler operated by Abonti LLC, a Russian technology company that provides a vertical search engine focused on e‑commerce and product listings. According to the official Abonti website (abonti.com), the bot is designed to index publicly accessible web pages — particularly product catalogs, pricing data, and inventory information — to feed its own search results and price‑comparison service. Unlike general‑purpose search engines, Abonti targets commercial content to enable users to compare prices across multiple online retailers.

🌐 Technical Behavior

Abonti crawls using a distributed set of IP addresses predominantly allocated to Russian autonomous systems (ASNs), such as AS208722 and AS197068, as documented in RIPE WHOIS records. The bot respects a default crawl delay of 5 seconds between requests unless otherwise specified in robots.txt. It uses HTTP/1.1 with persistent connections and sends a Referer header pointing to its own domain. Crawl frequency is moderate — typically a few hundred requests per hour per domain — but can spike when scanning large product feeds. The bot does not execute JavaScript or load external resources, focusing only on static HTML content.

📋 robots.txt Compliance

The Abonti crawler honors Disallow directives in robots.txt as stated on its official bot information page (abonti.com/bot.html). It also respects the Crawl-Delay directive. Third‑party audits (e.g., bot‑detection tools like CrawlerManager) confirm that Abonti consistently obeys these rules and does not ignore rate‑limiting instructions. However, misconfigured robots.txt files may still be overlooked if they contain syntax errors.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; Abonti/1.0; +http://www.abonti.com/bot.html). Earlier versions may appear as simply Abonti/1.0. Behavioral fingerprints include a consistent lack of Accept-Language headers and a fixed Connection: keep-alive. Additionally, the bot always includes a From header with an email address ([email protected]) in its requests, a practice that helps webmasters identify it.

📊 Data Usage

Collected data — including product names, prices, descriptions, and availability — is used exclusively to populate the Abonti search and price‑comparison engine. The company states that it does not repurpose content for AI training or sell aggregated data to third parties. According to their privacy policy, cached copies of crawled pages are retained for up to 30 days to facilitate fresh search results.

⚙️ Rate Limiting Policy

Abonti is rate‑limited because its focused indexing of e‑commerce pages can generate burst traffic when scanning large product catalogs. The policy rationale for threshold‑based blocking is to prevent resource exhaustion on server‑side databases and to ensure fair access for other legitimate crawlers. Webmasters are encouraged to enforce a moderate crawl‑delay (e.g., 10 seconds) if the default 5‑second interval still causes load issues.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.