🤖 Overview

DotBot is a web crawler operated by Moz (formerly SEOmoz), a leading provider of SEO software and analytics. It was first introduced in 2009 as part of Moz’s Open Site Explorer tool and continues to power the Link Explorer and other Moz SEO products. Its primary purpose is to systematically discover and analyze publicly accessible web pages to build a comprehensive index of hyperlinks, page metrics, and site structure for search engine optimization research.

🌐 Technical Behavior

DotBot crawls using standard HTTP/1.1 requests and identifies itself via the User-Agent string "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot)" or "Mozilla/5.0 (compatible; DotBot/1.2; http://www.moz.com/dotbot)". It typically fetches only the HTML content of a page and does not follow JavaScript or CSS files unless explicitly allowed. The bot respects the Crawl-Delay directive specified in robots.txt and can maintain a variable crawl rate depending on the site’s response time. Moz publishes a list of IP ranges used by DotBot, which includes subnets like 199.71.214.0/24 and 104.18.0.0/16 (as per Moz’s official documentation and WHOIS records). The crawler can be verified by performing a reverse DNS lookup on its IP address; valid DotBot hosts resolve to a domain ending in .moz.com. It does not send a custom X-Robots-Tag header but adheres to standard meta robots tags. DotBot does not cache or serve content to end users; it only stores link and metric data on Moz’s servers.

📋 robots.txt Compliance

Moz explicitly states that DotBot fully respects robots.txt directives, including Disallow, Allow, and Crawl-Delay. Webmasters can control DotBot’s access using standard robot exclusion rules. Evidence from Moz’s support pages and community forums confirms that the crawler checks robots.txt before every crawl request and will not access disallowed paths. However, because DotBot’s index is used for link analysis rather than content display, Moz advises that blocking it may reduce the accuracy of link metrics reported in Moz tools.

🔍 Detection Indicators

The primary detection method is the User-Agent string: "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot)" or "Mozilla/5.0 (compatible; DotBot/1.2; http://www.moz.com/dotbot)". Additionally, DotBot’s requests typically lack common browser headers such as Accept-Language or Accept-Encoding, and the IP reverse-lookup always produces a hostname like "crawl-xxx-xxx-xxx-xxx.moz.com". Behavioral fingerprints include a consistent crawl interval (usually every few seconds to minutes) and a preference for fetching only text/html responses. Moz provides a verification tool at https://moz.com/help/guides/moz-web-crawler/verify-dotbot for webmasters to confirm the identity of a suspected DotBot request.

📊 Data Usage

Data collected by DotBot is exclusively used to populate Moz’s Link Explorer index, which supplies metrics such as Domain Authority, Page Authority, Spam Score, and inbound link counts. The index is also leveraged by Moz’s MozBar browser extension, Moz Pro campaign tools, and the discontinued Open Site Explorer. No content is used for AI model training, advertising, or resale; the crawled data is limited to link structure and page-level metadata like title tags and meta descriptions. Moz periodically refreshes its index (approximately every two to four weeks) by re-crawling known pages and discovering new ones, ensuring link data remains current.

⚙️ Rate Limiting Policy

Because DotBot must crawl millions of pages to maintain a comprehensive link index, it can appear aggressive to servers with limited bandwidth. Rate-limiting is recommended to prevent resource exhaustion; Moz itself suggests using the Crawl-Delay directive in robots.txt to set a polite interval. Many web application firewalls and CDN services implement threshold-based blocking when DotBot’s request rate exceeds a defined limit (e.g., 10 requests per second), which is a reasonable practice to protect server performance without permanently denying access to the crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.