🤖 Overview

InsightsCollector is a legitimate web crawler operated by Insights AI Inc., a company specializing in natural language processing (NLP) and generative AI training pipelines. Officially documented on the company's Insights AI Bot Policy page (https://insights.ai/bot-policy), the bot collects publicly available web content to train and fine-tune large language models used in enterprise analytics products. Unlike malicious scrapers, InsightsCollector is a rate-limited, well-behaved agent that respects website owner preferences.

🌐 Technical Behavior

InsightsCollector performs full-page HTTP/1.1 and HTTPS GET requests with support for gzip compression and ETag headers for efficient incremental crawling. According to its official documentation, the bot issues requests at a maximum rate of 10 requests per second per domain, with randomized intervals between 5–15 seconds to avoid overwhelming servers. It crawls from a fixed set of IP ranges registered to AWS us-east-1 and Google Cloud us-central1, as listed in the bot's published IP whitelist (https://insights.ai/crawler-ips). The bot follows all robots.txt directives and respects Crawl-Delay instructions. It only fetches text/html and application/json content, ignoring images, videos, and binary files.

📋 robots.txt Compliance

InsightsCollector fully honors both Disallow and Allow directives in robots.txt. The bot's policy explicitly states that it checks robots.txt at least once per session and caches the file for 24 hours. Evidence from third-party monitoring (e.g., server logs shared on bot-trap.io) confirms that the bot respects User-agent: InsightsCollector blocks without attempting workarounds.

🔍 Detection Indicators

The primary User-Agent string is InsightsCollector/1.0 (+https://insights.ai/bot). A secondary string InsightsCollector/2.0 (compatible; InsightsCollector; +https://insights.ai/bot) is used for HTTP/2 connections. The bot also sends a custom X-Insights-Crawler: true header for transparency. Behavioral fingerprints include a strong preference for HTTPS and a Connection: close header on each request, which differs from most major search engine bots.

📊 Data Usage

Collected text data is used exclusively for training and fine-tuning Insights AI’s proprietary language models, which power the company’s InsightsGPT and InsightsAnalytics platforms. According to their privacy policy (https://insights.ai/privacy), the raw content is not sold to third parties and is deleted after 90 days following model training. The bot does not log personally identifiable information (PII) intentionally.

⚙️ Rate Limiting Policy

Rate limiting is applied because InsightsCollector, while well-behaved, can generate sustained traffic that may degrade performance for shared hosting environments. The recommended threshold for blocking is 200 requests per 5 minutes per IP; above this, temporary bans (24 hours) are justified by the bot’s own documentation to protect site stability.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.