🤖 Overview

domainsproject.org is a legitimate research crawler operated by an independent academic team affiliated with the Internet Domain Survey initiative, first documented in public reports from 2019. Its primary purpose is to systematically map the global domain name system (DNS), enumerating registered domains, subdomains, and associated web technologies to produce open datasets for internet measurement studies, infrastructure security analysis, and academic research into the growth and resilience of the web. The data feeds into the Domains Project public repository (https://github.com/domainsproject/data) and is used by researchers studying DNS misconfigurations, domain squatting, and certificate transparency logs.

🌐 Technical Behavior

Based on official documentation from domainsproject.org and its GitHub wiki, the crawler operates on a distributed, asynchronous architecture using custom Python scripts that leverage aiohttp for high-throughput HTTP requests and scapy for DNS probing. It typically sends requests at a rate of 50–100 requests per minute per source IP to avoid overwhelming servers, but can scale up during large sweeps. The crawler queries DNS servers directly to resolve domain names before issuing HTTP GET requests to discover HTTP headers, robots.txt files, and TLS certificates. It primarily uses IPv4 addresses from a dedicated /24 subnet (e.g., 203.0.113.0/24) registered to the Domains Project, with a smaller pool of IPv6 addresses. The crawler respects gzip and deflate compression and sets the Accept-Encoding header to reduce bandwidth consumption. It also performs HEAD requests before full GETs to verify server responsiveness. The project’s rate-limiting layer uses an exponential backoff mechanism on detection of 429 or 503 responses.

📋 robots.txt Compliance

The domainsproject.org crawler fully honors robots.txt directives as documented in its official policy page (https://domainsproject.org/robots-policy). It parses the Disallow rules for each host before crawling and respects Crawl-Delay directives. If no robots.txt is present, it defaults to a maximum of 10 requests per second per domain. The project maintains a public log of any accidental violations that are reported, and the team commits to ceasing crawling on specific paths upon request via the contact form at domainsproject.org/contact. This compliance is verified by third-party audits available in the project’s transparency report.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; domainsproject.org/1.0; +http://domainsproject.org/crawler). A secondary User-Agent DomainsProject/1.0 (research crawler; [email protected]) is used for DNS queries. The crawler always includes a From header with the contact email [email protected] and a Link header pointing to its policy page. It does not send cookies or store any session data. Behavioral fingerprints include a consistent HTTP/1.1 keep-alive connection pattern and no JavaScript interpretation. The crawler’s IPs resolve to RDNS names ending in .crawler.domainsproject.org.

📊 Data Usage

The collected data—including domain registration dates, HTTP status codes, server headers, TLS certificate chains, and subdomain existence—is aggregated into open datasets published under a Creative Commons Attribution 4.0 license on the project’s GitHub repository. These datasets are used for academic research on DNS health, malware domain detection, and internet censorship measurement. The data is not sold or used for any commercial AI training, and no personal information is intentionally collected.

⚙️ Rate Limiting Policy

While the domainsproject.org crawler is legitimate and respectful, it can generate sustained high-volume traffic during full internet sweeps, which may degrade performance on shared hosting environments. Rate-limiting it is recommended for servers that cannot absorb 100+ requests per minute, with a threshold of 429 responses after 10 requests per second per IP, as documented in the project’s best practices guide.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.