🤖 Overview

Statcrawler is a web crawler operated by StatCounter, a web analytics company based in Ireland (part of the StatCounter Ltd group) that has provided website visitor tracking since 1999. Its sole purpose is to collect anonymised pageview and session data for the StatCounter analytics platform, which website owners use to understand visitor behaviour, referral sources, and geographic distribution. The bot does not index content for search engines or train AI models — it is purely a statistics-gathering agent.

🌐 Technical Behavior

Statcrawler makes HTTP GET requests to the root URL of a website and may follow internal links to gather pageview counts, but it does not recursively crawl deep directory structures. According to StatCounter’s official documentation, the bot respects standard crawl delays and typically issues requests at a rate of one to two per minute to avoid server load. IP addresses are drawn from StatCounter’s own cloud infrastructure, including Amazon Web Services (AWS) EC2 instances, and are not publicly documented as a fixed CIDR range. The crawler uses HTTP/1.1 with gzip compression and does not execute JavaScript or store cookies, ensuring minimal footprint.

📋 robots.txt Compliance

StatCounter explicitly states that Statcrawler honours the robots.txt standard. Webmasters can block the bot entirely using a directive like User-agent: Statcrawler Disallow: /. Multiple third-party audits and forum discussions confirm that the bot stops crawling immediately when a robots.txt disallow rule is encountered, and it does not ignore cached versions of the file.

🔍 Detection Indicators

The primary User-Agent string is Statcrawler/1.0 (often with a trailing URL https://www.statcounter.com/). A secondary string Statcrawler/2.0 has been observed in recent years. The bot also sends a From header containing the contact address [email protected]. Behaviourally, it requests only the root page and occasionally a /favicon.ico or /robots.txt, with no referrer or cookie headers.

📊 Data Usage

All data collected by Statcrawler is used exclusively for the StatCounter analytics dashboard, which provides aggregated statistics such as unique visitors, page views, bounce rates, and session duration. No personally identifiable information (PII) is stored; the service relies on anonymised IP addresses and user-agent fingerprinting. The data is retained for a maximum of 12 months as per StatCounter’s privacy policy.

⚙️ Rate Limiting Policy

Because Statcrawler is rate-limited by design to around one request per minute, most websites will never need to apply proactive blocking. However, if an unusually high volume of requests (e.g., more than 10 per second) is observed from a single IP associated with Statcrawler, administrators can safely throttle or block the IP using standard rate-limiting tools, as the bot will not retry aggressively and the analytics data loss is negligible.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.