🤖 Overview

hitcrawler is a legitimate web crawler operated by Hitwise, a subsidiary of Connexity (formerly a division of Experian), designed to collect publicly available web traffic data and site content for competitive digital market intelligence. Its primary purpose is to feed the Hitwise platform, which provides analytics on website rankings, search trends, and consumer behavior to marketers and enterprises. The bot has been active since at least 2005 and is explicitly documented in Hitwise’s official support pages as a required component for their data collection methodology.

🌐 Technical Behavior

hitcrawler employs a distributed crawling architecture that issues HTTP GET requests to thousands of domains daily, typically at a moderate rate of 1–2 requests per second per IP to avoid overwhelming servers. Its crawl patterns focus on publicly accessible pages such as homepages, category pages, and sitemaps, with a strong preference for sites that are popular or high-traffic. The bot uses a standard HTTP/1.1 protocol and does not log in or submit forms; it only fetches public content. IP ranges used by hitcrawler are documented in official Hitwise documentation as belonging to netblocks assigned to the company, including IPs from the 23.XX.XX.X and 208.XX.XX.X ranges, and are often listed on public IP allowlists maintained by CDN providers like Cloudflare. The crawler respects the If-Modified-Since header to reduce bandwidth usage and typically operates during business hours in the target market’s time zone.

📋 robots.txt Compliance

According to Hitwise’s published guidelines, hitcrawler fully honors robots.txt directives and will cease crawling any URL or path that is explicitly disallowed. Evidence from multiple site administrators’ forums confirms that adding a Disallow: / for the hitcrawler User-Agent stops all requests immediately, and the bot does not attempt to circumvent such rules. Hitwise encourages site owners to use robots.txt to control access rather than IP blocking, and they maintain a dedicated email address ([email protected]) for escalation.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; hitcrawler/1.0; +http://www.hitwise.com/uk/about-us/botpolicy.html), which is publicly verifiable on the official Hitwise bot policy page. Behavioral fingerprints include a consistent request interval of 2–5 seconds between pages, a lack of JavaScript execution, and the absence of HTTP headers like Accept-Language or Referer. The bot also sends a User-Agent header that always contains the substring “hitcrawler”.

📊 Data Usage

Data collected by hitcrawler is used exclusively to populate the Hitwise Digital Consumer Intelligence platform, which aggregates anonymous, aggregated web traffic patterns and search behavior for competitive benchmarking. No personal or identifiable information is stored; the platform focuses on ISP-level clickstream data and public site content to generate reports on market share, keyword trends, and audience profiles. Hitwise explicitly states they do not sell raw data or use it for AI training—only for aggregated analytics.

⚙️ Rate Limiting Policy

hitcrawler is rate-limited because its sustained request patterns can be mistaken for aggressive scraping by security tools, even though its rate is low. Threshold-based blocking is recommended when the crawler exceeds 10 requests per second (which is rare) or if it ignores robots.txt directives for more than 24 hours—both scenarios warrant a temporary block to safeguard server resources.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.