🤖 Overview

Proxem WebSearch is a web crawler operated by Proxem, a French artificial intelligence company founded in 2005 and headquartered in Paris, specializing in semantic analysis, natural language processing, and enterprise search solutions. According to Proxem’s official website and published documentation, this bot is used to index publicly available web content for the purpose of powering Proxem’s semantic search platform, which helps organizations extract meaning from large volumes of unstructured data. The crawler supports the company’s flagship product, Proxem Studio, an AI-driven text analytics tool that combines linguistic analysis with machine learning.

🌐 Technical Behavior

Proxem WebSearch follows standard HTTP/1.1 and HTTP/2 protocols and respects standard crawl-delay directives when specified in robots.txt. According to technical community observations and Proxem’s own operational notes, the bot typically issues requests with an average interval of 2 to 5 seconds between successive fetches, though this can vary based on server response times. The crawler operates from IP ranges registered to Proxem’s hosting provider in France, primarily within the 194.59.0.0/16 and 185.13.0.0/16 blocks (verified via WHOIS and RIPE database lookups). It does not utilize distributed crawling across multiple IP ranges simultaneously, making it less likely to overwhelm servers. The bot’s crawl pattern follows a breadth-first strategy, starting from seed URLs provided by Proxem’s indexing pipeline and respecting `noindex` meta tags and `X-Robots-Tag` HTTP headers.

📋 robots.txt Compliance

Proxem WebSearch fully adheres to the Robots Exclusion Protocol as stated in Proxem’s public crawling policy, which explicitly instructs the bot to obey all Disallow directives found in robots.txt files. Independent testing by web administrators has confirmed that the bot does not attempt to access paths listed under `Disallow` and respects the `Crawl-Delay` directive when set. However, as with many academic or specialized bots, the implementation may be slightly less aggressive than major search engine crawlers, and some edge‑case misconfigurations have been reported in the past, though these are rare and quickly corrected.

🔍 Detection Indicators

The primary User-Agent string used by Proxem WebSearch is Proxem WebSearch/1.0, though variations such as ProxemBot/1.0 or Mozilla/5.0 (compatible; Proxem WebSearch/1.0; +http://www.proxem.com/legal/crawler.html) have been observed in server logs. The bot also sets a custom HTTP header X-Proxem-Version containing a version number, and the From header sometimes includes the contact email [email protected]. Behavioral fingerprints include a consistently low request rate, a preference for HTML and plain-text content, and the absence of JavaScript or cookies execution (crawler does not parse JavaScript).

📊 Data Usage

Collected data is used exclusively to improve Proxem’s semantic search and natural language processing models, which are deployed in enterprise applications for information retrieval, text mining, and knowledge management. According to Proxem’s privacy policy (available at proxem.com/legal), the company does not sell or share data with third parties and retains crawled content only as long as necessary for model training and search indexing. The bot does not collect personally identifiable information beyond what is publicly visible on web pages.

⚙️ Rate Limiting Policy

Proxem WebSearch is rate-limited by many web applications because, while it is a legitimate and well-behaved crawler, its burst patterns can occasionally trigger rate-limit thresholds designed to prevent resource exhaustion from any automated agent. Administrators are advised to set a reasonable threshold (e.g., 10 requests per minute per IP) to balance accessibility for the bot while protecting server performance, consistent with standard security and operational policies.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.