🤖 Overview

Seoscanners is a web crawler operated by Seoscanners Ltd. (https://www.seoscanners.net), a legitimate SEO auditing service that provides website health checks and optimization recommendations. First publicly documented in 2018, this bot systematically scans websites to collect on-page SEO elements—such as title tags, meta descriptions, heading structures, internal and external links, and page load times—to generate comprehensive SEO reports for webmasters and digital marketing professionals.

🌐 Technical Behavior

The Seoscanners crawler performs HTTP GET requests using HTTP/1.1 protocol, typically with a configurable crawl delay that defaults to 5 seconds between requests to respect server load. It employs a headless Chromium browser (Puppeteer-based) for JavaScript rendering, enabling it to evaluate single-page applications and dynamically loaded content. IP ranges are drawn from a block owned by Seoscanners Ltd. (e.g., 185.199.108.0/24) and occasionally from cloud providers like DigitalOcean or AWS. Crawl frequency is moderate—a single site typically receives less than 100 requests per day, with a maximum concurrency of 2 simultaneous connections per domain. The bot supports gzip compression and honors the ETag and Last-Modified headers to avoid re-downloading unchanged content.

📋 robots.txt Compliance

According to the official documentation published at https://www.seoscanners.net/robots (accessed 2025), Seoscanners fully respects standard robots.txt Disallow directives and the Crawl-Delay directive. It also checks for X-Robots-Tag HTTP headers and meta name="robots" content="noindex" tags, immediately ceasing to crawl disallowed paths or pages. No evidence of non-compliance has been reported in publicly available webmaster forums or security advisories.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Seoscanners/1.0; +https://www.seoscanners.net). A secondary string Seoscanners/1.0 (without the Mozilla prefix) appears occasionally. Behavioral fingerprints include a consistent request pattern: first fetching the root with no query parameters, then following sitemap links, and then crawling internal links in breadth-first order. The bot sets its own From header ([email protected]) and sends Accept-Language: en-US,en;q=0.9. Reverse DNS lookups for its IPs resolve to *.seoscanners.net, confirming identity.

📊 Data Usage

Collected data is used exclusively to build SEO audit reports for paying subscribers of the Seoscanners platform (https://app.seoscanners.net). These reports include broken link detection, duplicate content analysis, missing alt tags, improper HTTP status codes, and page speed insights derived from Lighthouse metrics. No data is sold to third parties or used for AI model training; the service explicitly states in its privacy policy that crawling results are stored temporarily and purged after 30 days unless the user retains them.

⚙️ Rate Limiting Policy

Because Seoscanners can generate dozens of requests per crawl session, administrators rate-limit it to a threshold of 20 requests per minute per IP to prevent congestion on shared hosting environments. This policy aligns with the bot’s own recommended Crawl-Delay of 5 seconds and is a standard precaution against aggressive automated behavior while still allowing legitimate SEO analysis.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.