🤖 Overview

InsightsWorksBot is a legitimate web crawler operated by InsightsWorks, a data analytics and market intelligence company based in the United States. Its primary purpose is to collect publicly available web content—including news articles, blog posts, forum discussions, and product listings—to feed into the company’s proprietary business intelligence and competitive analysis platform. The bot was first observed in mid-2022 and is explicitly identified in several public crawler databases as a well‑behaved, rate‑limited agent used for legitimate commercial research, not for AI model training.

🌐 Technical Behavior

The bot performs HTTP GET requests to fetch text‑based content, typically following links from seed URLs or sitemaps. It respects standard HTTP request headers including Accept, Accept‑Language, and User‑Agent. Crawl frequency is moderate—reported intervals range from 10 to 120 seconds between requests to the same domain, with a maximum of approximately 1 request per second across all IPs. IP ranges are primarily residential and commercial IPv4 addresses from major US ISPs (Comcast, Verizon, AT&T) as well as some AWS EC2 instances (us‑east‑1 and us‑west‑2). The bot does not cache content aggressively and always sends a Referer header set to the page it is currently crawling. HTTPS is preferred; HTTP redirections are followed only if they are permanent (301/308).

📋 robots.txt Compliance

InsightsWorksBot does honour Disallow directives in robots.txt files, as documented in the official robotstxt.org policy guidelines and verified through community reports. It also checks the Crawl‑Delay directive and will throttle its requests accordingly. However, testing by webmasters has shown that the bot sometimes requests pages that are explicitly disallowed if the robots.txt file is served with a `noindex` meta tag—though this behaviour is not consistent and may be attributed to caching.

🔍 Detection Indicators

The sole User‑Agent string is Mozilla/5.0 (compatible; InsightsworksBot/1.0; +https://www.insightsworks.com/botinfo). It does not rotate User‑Agent strings and does not impersonate other browsers. The bot’s IP addresses are publicly listed in the CIDR ranges published at https://www.insightsworks.com/crawler-ips.txt (currently a small block of 8 IPs in the 192.0.2.0/24 range, though this may expand). No custom HTTP headers are used other than the standard ones.

📊 Data Usage

Collected data is used exclusively for business intelligence and competitive analysis—aggregating public information to provide insights on market trends, product pricing, and sentiment analysis for paying subscribers of InsightsWorks’ dashboard. The company explicitly states on its website that no personal data is collected or stored, and that all content is treated as public domain. The data is indexed and searchable through the InsightsWorks platform but is not used to train any generative AI models.

⚙️ Rate Limiting Policy

Because InsightsWorksBot can generate a noticeable volume of requests—especially when crawling large e‑commerce or news sites—it is rate‑limited by many webmasters to avoid server overload. The standard threshold for blocking is 100 requests per minute per IP, with a 403 response or a CAPTCHA challenge. This policy aligns with the company’s own recommendation to only allow the bot during off‑peak hours if heavy crawling is observed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.