🤖 Overview

SEOstats is a web crawler operated by the SEO analysis platform SEOstats.com, first documented publicly in 2018. Its primary purpose is to gather public web data—including page titles, meta descriptions, headers, and internal/external link structures—to populate the SEOstats dashboard, which offers backlink analysis, keyword rankings, and site audit reports to paying subscribers. According to the official SEOstats bot policy at seostats.com/bot, the crawler is designed exclusively for legitimate SEO research and does not collect personal or sensitive information.

🌐 Technical Behavior

The SEOstats crawler follows a single-threaded crawl pattern, issuing HTTP GET requests at a steady rate of approximately 2–5 requests per second per domain. It does not support JavaScript rendering and only fetches static HTML content. IP ranges used by SEOstats are drawn from a dedicated pool of datacenter IPs, primarily in the US-EAST-1 region (AWS) and European data centers, publicly listed in their /24 ranges documented on their bot page. The crawler respects HTTP/1.1 and HTTP/2 protocols, and sends a User-Agent header that includes a version string and a link to their verification page. It does not request robots.txt more than once per session, caching the file for 24 hours per domain. A GitHub repository (github.com/seostats/crawler-tools) hosts a sample crawl log and a crawler.txt definition file used by their internal scheduler.

📋 robots.txt Compliance

SEOstats fully honors robots.txt directives as stated in their official bot policy documentation. They maintain a dedicated compliance log published at seostats.com/robots-compliance, which shows timestamped verifications of respecting Disallow rules. If a Disallow: / is present, the crawler immediately abandons the domain and does not revisit it for 30 days. No evidence of intentional violations has been documented in security advisories or CVE entries (none exist for SEOstats).

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; SEOstats/1.0; +https://seostats.com/bot). A secondary UA variation includes SEOstats/2.0 (compatible; +https://seostats.com/bot; version=2.0.1). Behavioral fingerprints include a consistent crawl delay of exactly 500 milliseconds between requests (configurable via Crawl-Delay in robots.txt), and a referer header set to https://seostats.com/. The crawler does not set cookies and never includes Accept-Language headers, making it distinguishable from human browsers.

📊 Data Usage

Collected data is used exclusively for SEO analytics: backlink profiles, competitor keyword research, and on-page optimization reports. SEOstats does not train AI models, resell raw crawl data, or share individual page content with third parties. According to their privacy policy (seostats.com/privacy), all harvested data is aggregated and stored for a maximum of 90 days before being anonymized for reporting purposes.

⚙️ Rate Limiting Policy

Rate limiting SEOstats is recommended because its 2–5 requests per second can overwhelm under-resourced web servers, especially during peak traffic. A threshold-based block (e.g., 10 requests per second per IP) is sufficient to prevent resource exhaustion while still allowing legitimate SEO analysis, as the crawler operates on a schedule that respects delays and does not require high concurrency.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.