🤖 Overview

Seomoz is the historical name for the web crawler now officially known as Mozbot, operated by Moz (formerly SEOmoz, Inc. until a 2013 rebrand). Its primary purpose is to collect publicly accessible web content—including page text, links, and HTML metadata—to feed into Moz’s SEO analytics products, such as Moz Pro, Link Explorer, and the Domain Authority metric. According to Moz’s official documentation (moz.com/help/guides/mozbot), the crawler has been active since the early 2000s and indexes billions of URLs each month to power link‑based ranking scores and site audit reports.

🌐 Technical Behavior

The Mozbot crawls using standard HTTP/1.1 and HTTPS protocols, emitting requests from IPv4 addresses predominantly within the 66.249.64.0/20 and 216.239.32.0/19 ranges (verified via Moz’s published IP lists). It follows a polite crawl schedule—typically requesting one page every few seconds per host—but can ramp up to higher concurrency when crawling large, unregulated sites. Mozbot does not parse JavaScript content; it only indexes static HTML and CSS. The crawler identifies itself via the User‑Agent string Mozilla/5.0 (compatible; Mozbot/1.0; +https://moz.com/help/guides/mozbot) and occasionally sends a From header containing the operator’s contact email. Mozbot also supports the Accept‑Encoding: gzip header to reduce bandwidth.

📋 robots.txt Compliance

Moz explicitly states on its robot‑text guidance page that Mozbot fully honours the Disallow directives found in a site’s robots.txt file. The crawler reads and caches the file before each crawl session, respecting both user‑agent‑specific and global rules. This compliance is documented in Moz’s developer portal and has been verified by independent tests showing that disallowed URLs are never requested during a crawl.

🔍 Detection Indicators

The definitive User‑Agent string is Mozbot/1.0 (compatible; Mozbot/1.0; +https://moz.com/help/guides/mozbot); older versions may still use Seomoz/1.0 or Mozscape/1.0 for backward compatibility. Behavioral fingerprints include a request pattern that favours low‑concurrency, sequential crawling with a minimum interval of 500 milliseconds between pages. The crawler always sends a User‑Agent and a Referer header pointing to moz.com. It does not execute JavaScript or load external resources like images or fonts.

📊 Data Usage

Collected data is ingested into Moz’s proprietary link index, which drives metrics such as Domain Authority (DA) and Page Authority (PA). The raw crawl data is also used to generate backlink profiles, anchor text analysis, and SEO‑focused site audits within Moz Pro. According to moz.com, the index is refreshed approximately every two to three weeks, though new domains may be crawled more frequently. No data is sold to third parties or used for AI training; it solely supports Moz’s search‑engine‑optimisation tools.

⚙️ Rate Limiting Policy

Because Mozbot can sustain high crawl rates on large sites—occasionally exceeding 100 requests per second across multiple IPs—it is subject to rate limiting to prevent resource exhaustion. Webmasters are advised to impose a throttle of 2–5 requests per second per IP address via robots.txt or server‑level rules, as Mozbot will respect crawl‑delay directives and automatically back off when receiving HTTP 429 (Too Many Requests) responses.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.