🤖 Overview

SafeDNSBot is a legitimate web crawler operated by SafeDNS, a UK-based cybersecurity company specializing in DNS filtering, web content categorization, and threat intelligence. According to SafeDNS’s official documentation at https://www.safedns.com/robots.txt and their bot policy page, this bot is primarily used to scan and classify websites into over 60 content categories (e.g., adult, gambling, social media, malware) to feed SafeDNS’s cloud-based content filtering and parental control services. The bot also helps maintain a real-time database of domain reputation for SafeDNS’s blocklist and allowlist systems, which are used by ISPs, schools, and enterprises to enforce web usage policies.

🌐 Technical Behavior

SafeDNSBot performs HTTP GET requests at a moderate crawl rate, typically limited to a few requests per second per domain to minimize server load. The bot respects crawl delays specified in robots.txt and usually waits at least 5 seconds between consecutive requests. Its IP addresses are drawn from a dedicated range assigned to SafeDNS, which can be verified via reverse DNS lookups of the format bot-*.safedns.com. Crawl patterns are deterministic: the bot requests a site’s homepage, sitemap.xml, and common paths like /robots.txt first, then follows internal links up to a configurable depth (typically 4–5 levels). SafeDNSBot supports both IPv4 and IPv6 and uses a persistent connection pool to improve efficiency. According to SafeDNS’s engineering blog, the crawler operates 24/7 but throttles itself during peak hours on small websites to avoid disruption.

📋 robots.txt Compliance

SafeDNSBot is documented to fully obey robots.txt directives, including User‑agent, Disallow, and Crawl‑delay instructions. SafeDNS publicly states that their crawler will never access paths explicitly blocked and will respect a site’s robots.txt file as a condition of its operation. In practice, independent security researchers have confirmed that SafeDNSBot does not attempt to bypass disallowed directories, making it one of the more compliant crawlers in the content classification space.

🔍 Detection Indicators

The primary User‑Agent string for SafeDNSBot is “SafeDNSBot/1.0 (+https://www.safedns.com/bot)” (case‑sensitive). Variations include “SafeDNSBot/2.0” for the updated crawler released in 2024. Additional identifying headers include a From: [email protected] email header in initial requests, and a custom X‑SafeDNS‑Crawl: 1 header to assist server administrators. The bot’s IP ranges are published at https://www.safedns.com/ip‑ranges.txt, which currently lists 8.28.16.0/23 and 104.16.0.0/12 (Cloudflare‑backed) — the latter are proxy IPs, so direct detection via reverse DNS is recommended.

📊 Data Usage

Collected data—page titles, meta descriptions, HTML content, and domain metadata—is used exclusively to classify websites into SafeDNS’s proprietary category taxonomy and to update domain reputation scores in near real‑time. Classification results are served to SafeDNS’s filtering DNS servers, which block or allow content according to user policies. SafeDNS states they do not store raw page content beyond 30 days, nor do they use the data for AI training, advertising, or resale to third parties (as per their privacy policy at https://www.safedns.com/privacy). The dataset also powers their threat intelligence feed, which detects newly registered domains hosting malware or phishing.

⚙️ Rate Limiting Policy

SafeDNSBot is rate‑limited because its sustained crawling—though moderate—can still overload small sites or shared hosting environments if left unchecked. The recommended threshold is to allow up to 5 requests per second from its IP ranges, and to block or temporarily throttle if the bot exceeds that rate, as documented in SafeDNS’s own rate‑limiting guidelines for network administrators.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.