🤖 Overview

dkimrepbot is a legitimate web crawler operated by DKIM Reputation (dkim-reputation.org), an open-source project that aggregates email authentication data to provide domain reputation scores for email senders. Its primary purpose is to collect publicly available DMARC, DKIM, and SPF records from DNS and web pages, as well as mailing list archives and email headers, to build a comprehensive database used by email security systems to assess sender trustworthiness. The bot was first documented in 2018 and is maintained by a community of email security researchers.

🌐 Technical Behavior

dkimrepbot crawls web pages that contain email authentication records, such as DMARC policies, DKIM public keys embedded in HTML, and SPF TXT records published on websites. It uses HTTP GET requests over IPv4 and IPv6, with a default crawl frequency of approximately one request per 10 seconds per domain, though it may increase if a site returns many relevant pages. The bot originates from IP ranges assigned to cloud infrastructure used by DKIM Reputation, typically from Amazon Web Services and DigitalOcean, as listed in the official GitHub repository at github.com/dkim-reputation/dkimrepbot-ips. It does not follow JavaScript redirects and only parses plain HTML and DNS responses. The crawler identifies itself via the User-Agent string and also sends a custom HTTP header X-Robots-Tag (if present) to respect indexing directives. It caches DNS results for up to one hour to reduce load on authoritative servers.

📋 robots.txt Compliance

According to the official DKIM Reputation website, dkimrepbot fully respects the robots.txt exclusion standard. It will not crawl any page or directory that is disallowed by the site's robots.txt file, and it also honors the X-Robots-Tag HTTP header if set to "noindex". The bot checks robots.txt at the start of each crawl session and caches the rules for up to 24 hours. There are no known instances of the bot violating these directives.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: Mozilla/5.0 (compatible; dkimrepbot/1.0; +https://dkim-reputation.org/bot). Additionally, the bot may include a From header containing a contact email address (e.g., [email protected]). It does not send any cookies or session data, and its requests typically have a low Accept-Language value. The bot's IP addresses can be looked up via the project's public IP list published at their GitHub repository (github.com/dkim-reputation/dkimrepbot-ips). Behavioral fingerprints include a consistent request pattern of exactly one HTTP GET per page, no referrer header, and a default request timeout of 30 seconds.

📊 Data Usage

Collected data is used to populate the DKIM Reputation database, which provides domain reputation scores, DMARC compliance reports, and DKIM key validation results. This data is made available to email security vendors and network operators through a REST API and is used to improve email deliverability and detect phishing or spoofing attempts. No personally identifiable information is collected; only publicly accessible DNS records and email header metadata are processed. The aggregated reputation data is also used in academic research on email authentication effectiveness.

⚙️ Rate Limiting Policy

dkimrepbot is rate-limited because its scanning pattern, while legitimate, can generate a high volume of requests against DNS servers and web hosts, potentially impacting performance if left unchecked. A conservative threshold-based blocking approach ensures that the bot does not overwhelm smaller sites while still allowing it to gather necessary authentication data; this policy is documented in the project's guidelines at dkim-reputation.org/rate-limiting.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.