🤖 Overview

xpymep1.exe is a web crawler operated by Xpymep Ltd., a data analytics firm headquartered in Tallinn, Estonia, first publicly acknowledged in October 2022. Its primary purpose is to collect publicly accessible web content for training proprietary natural language processing models and for building a specialized vertical search index focused on technology and business publications. The bot feeds data into the company’s Xpymep Semantic Engine, a product that provides contextual search and content summarization APIs for enterprise clients.

🌐 Technical Behavior

The crawler uses a custom HTTP client implemented in C++ and typically issues requests at a sustained rate of 8–12 requests per second per source IP, as documented in the official operational notes at https://xpymep.com/bot. It employs a distributed architecture with IP ranges from 185.56.0.0/22 and 91.239.0.0/16, and rotates resolved hostnames across these blocks to avoid rate‑limiting. The bot sends standard HTTP headers including Accept‑Encoding: gzip, deflate and a unique X‑Bot‑Id header that varies per crawl session. It also respects the Crawl‑Delay directive and pauses after encountering 429 Too Many Requests responses for a configured backoff period.

📋 robots.txt Compliance

According to Xpymep’s publicly posted robots.txt policy at https://xpymep.com/robots.txt, the bot honors all Disallow directives and can be blocked entirely with the rule User‑agent: xpymep1.exe. Multiple website owner reports confirm that it does not ignore Disallow rules, though it may occasionally re‑crawl delayed paths after the directive’s scheduled refresh.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; xpymep1.exe; +https://xpymep.com/bot). Secondary identifiers include the X‑Bot‑Id header (format XP‑[hex‑16]) and a consistent request ordering that prioritizes robots.txt then sitemap.xml before other pages. The bot also sends a non‑standard Via header with the value xpymep‑crawler/1.0 in some cases, as noted in GitHub issue #1241 of the crawler-detection repository.

📊 Data Usage

Collected data is used exclusively to train Xpymep’s XSeq‑NLP models and to populate the vertical search index of the Xpymep Semantic Engine. The company’s privacy policy states that no raw HTML is shared with third parties, and all data is anonymized before being ingested into training pipelines. The index is refreshed weekly to maintain freshness.

⚙️ Rate Limiting Policy

Because the bot’s persistent crawling can generate significant load on smaller servers, rate limiting is advisable. Threshold‑based blocking (e.g., more than 20 requests per second from a single IP) is a reasonable defense to protect server resources without completely denying access to the legitimate crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.