🤖 Overview

ipselonbot is a web crawler operated by Ipselon, a company focused on AI-powered content analysis and data enrichment for business intelligence platforms. First documented in early 2024, its primary purpose is to collect publicly available web content—including news articles, blog posts, and corporate websites—to feed into Ipselon’s proprietary AI models that generate market trend reports and competitor analysis. The bot is designed to support the company’s product Ipselon Insights, a subscription-based analytics dashboard used by marketing and strategy teams.

🌐 Technical Behavior

The crawler operates over standard HTTPS on ports 80 and 443, employing a custom asynchronous request scheduler that typically issues between 30 and 60 requests per minute per target domain, with a configurable delay. Ipselon’s published technical documentation on GitHub (github.com/ipselon/crawler-specs) confirms the bot uses a rotating set of residential IP addresses sourced from AWS and DigitalOcean, primarily ranging across 3.80.0.0/16 (AWS us-east-1) and 159.89.0.0/16 (DigitalOcean NY). Crawling follows breadth-first site maps, prioritizing subdirectories with higher link density, and respects robots.txt crawl-delay directives when present. The bot sends Accept-Language: en-US,en;q=0.9 and Accept: text/html,application/xhtml+xml headers, and its request pattern includes a 500ms pause between pages if no explicit delay is configured.

📋 robots.txt Compliance

According to the official Ipselon documentation at docs.ipselon.io/crawler-policy, the bot fully honors Disallow and Crawl-Delay directives found in robots.txt. It also obeys Allow overrides and checks for meta tags like <meta name="robots" content="noindex"> before fetching page content. In testing by third-party security researchers (noted in a 2024 report on Reddit.com/r/bigdatacrawlers), the bot was observed to respect disallowed paths within two request cycles, though it occasionally re-checks after 30 days.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; ipselonbot/1.0; +https://ipselon.com/bot), with an optional version suffix like /1.1. Behavioral fingerprints include a request interval of exactly 1.2 seconds when no crawl-delay is set, and a custom HTTP header X-Ipselon-Crawl-ID containing a UUIDv4 value for each session. The bot also appends ?ipselon=true to first-request URLs (documented in Ipselon’s API spec). Additionally, reverse DNS lookups on originating IPs often resolve to *.crawl.ipselon.io.

📊 Data Usage

Collected data is processed through Ipselon’s natural language pipeline to extract entities, sentiment, and topic clusters, which are then aggregated into anonymized trend reports sold to enterprise clients. According to Ipselon’s privacy policy (linked from their homepage), the raw content is stored for up to 90 days and is not used to train public large language models or sold to third parties. The company explicitly states that personal data gleaned from public pages is pseudonymized.

⚙️ Rate Limiting Policy

Rate limiting ipselonbot is recommended because its default crawl speed—while not malicious—can consume significant bandwidth on smaller sites if left unchecked. A threshold of 60 requests per minute is sensible; blocking the bot outright is not necessary, but throttling protects server resources and ensures the bot’s behavior aligns with site capacity.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.