🤖 Overview

DomainScan is a legitimate web crawler operated by DomainScan.com, a domain monitoring and intelligence service. Its primary purpose is to systematically collect publicly available data from websites—including SSL certificate details, HTTP headers, DNS records, and WHOIS expiry dates—to feed into a subscription-based alert platform that notifies users of changes to their domains or third-party assets.

🌐 Technical Behavior

The crawler performs periodic, stateful scans over HTTP/1.1 and HTTPS, typically initiating requests from a fixed set of IPv4 addresses belonging to the 192.241.144.0/20 and 165.227.96.0/19 ranges, as documented in the official DomainScan bot FAQ (domainscan.com/bot). It follows a depth-first traversal of discovered links up to 3 levels deep and respects server-side rate limiting by introducing random delays of 2–10 seconds between requests. The bot strictly adheres to the Transfer-Encoding: chunked header and does not send cookies or execute JavaScript, making its fingerprint consistent and low-overhead.

📋 robots.txt Compliance

DomainScan fully supports and honours all Disallow directives found in a site’s robots.txt file, as confirmed by its published crawler policy (domainscan.com/robots). It will also cease crawling upon encountering a 403 or 410 status code, and its indexer does not re-crawl disallowed paths for up to 30 days unless the Disallow rule is removed.

🔍 Detection Indicators

The most reliable detection string is User-Agent: DomainScan/1.0 or Mozilla/5.0 (compatible; DomainScan/1.0; +https://domainscan.com/bot). Additional headers include X-Robots-Tag: noindex on embedded content and a custom From: [email protected] header for administrative contact. Behavioral fingerprints include requesting only HTML pages, avoiding binary files, and sending a single Accept-Language: en header.

📊 Data Usage

Collected data is aggregated into a domain‑watch dashboard that triggers alerts for SSL certificate expiry (e.g., 30 days before), WHOIS changes, DNS record modifications, and unexpected HTTP header shifts. The service does not train language models or sell raw crawl data; it solely powers a monitoring and alerting product aimed at DevOps, security teams, and domain investors.

⚙️ Rate Limiting Policy

Despite its legitimate nature, DomainScan can generate hundreds of requests per hour per domain during initial backfill scans, which justifies rate‑limiting thresholds (e.g., 50 requests/min) to prevent server degradation. The policy rationale is to allow the monitoring service to function without compromising site performance for human visitors.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.