🤖 Overview

DomainCrawler is a legitimate web crawler operated by DomainCrawler LLC, a company headquartered in Toronto, Canada, that provides domain intelligence and website analytics services. Its primary purpose is to systematically collect publicly available data from domain registration records, webpage content, and DNS metadata to feed into its commercial domain research platform (domaincrawler.com).

🌐 Technical Behavior

DomainCrawler employs a breadth‑first crawling strategy, starting from domain names discovered through public WHOIS databases, Certificate Transparency logs, and DNS zone files. It sends HTTP GET requests at an average rate of 2 requests per second per source IP, with bursts of up to 5 requests per second during initial domain discovery. The crawler originates from a documented IPv4 range: 198.50.0.0/18 (AS16276, OVH) and uses a rotating set of IPv6 addresses from the 2607:5300::/32 block. It supports HTTP/1.1, TLS 1.2 and 1.3, and honors gzip/deflate content‑encoding. DomainCrawler does not render JavaScript or execute browser‑side code, relying solely on static HTML extraction and HTTP response headers.

📋 robots.txt Compliance

According to its official policy published at domaincrawler.com/robots.txt, DomainCrawler explicitly respects robots.txt directives. It checks the file once per domain and caches it for 24 hours, applying both Disallow rules and Crawl‑delay values when present. Independent audits by bot detection services (e.g., botchecker.io) confirm that DomainCrawler consistently obeys disallowed paths in real‑world testing.

🔍 Detection Indicators

The primary User‑Agent string is "DomainCrawler/1.0 (+http://domaincrawler.com/)". Older versions also use "DomainCrawler/0.9 (compatible; MSIE 6.0;)" in certain IPv4 ranges. Additional identifiers include a custom HTTP header X‑DomainCrawler: true and a From header containing [email protected] for contact purposes. The crawler always includes a non‑empty Referer field pointing to its own site.

📊 Data Usage

Collected data is aggregated into a searchable index of domain metadata (creation date, registrar, nameserver history), website technology stacks, and content summaries. This index is sold to SEO professionals, domain investors, and cybersecurity researchers for purposes such as competitive analysis, expired domain discovery, and DNS reconnaissance. DomainCrawler also provides a free public API (rate‑limited to 100 queries/day) for verifying domain attributes.

⚙️ Rate Limiting Policy

DomainCrawler is rate‑limited because its systematic, multi‑threaded crawling can generate traffic spikes that risk degrading server performance for other users. A threshold‑based blocking policy (e.g., more than 10 requests per second from a single IP) is recommended by webmasters to maintain service stability while still allowing legitimate access to the crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.