🤖 Overview

Page Analyzer is a web crawling agent operated by Moz (formerly SEOmoz), a well‑known SEO software company. First publicly documented in 2011, its primary purpose is to collect page‑level data for Moz’s “Page Analysis” and “Site Crawl” tools, which help website owners audit on‑page SEO elements such as title tags, meta descriptions, headings, and internal link structures. The bot feeds into Moz’s proprietary Link Explorer and Keyword Explorer products, enabling users to track rankings and identify technical issues.

🌐 Technical Behavior

The crawler follows a breadth‑first crawl strategy, typically visiting URLs found in sitemaps or linked from previously crawled pages. It sends requests at intervals of 2–5 seconds per domain to respect server load, as documented in Moz’s official developer guidelines. The bot primarily uses HTTP/1.1 and supports both IPv4 and IPv6. IP ranges are publicly listed in Moz’s help center under the “Crawler IP Addresses” section, with addresses belonging to Moz’s AWS‑based infrastructure (e.g., 54.236.1.xx range). It fetches only HTML content and explicitly ignores JavaScript, CSS, and images unless needed for accessibility analysis. The crawler sets the Accept‑Encoding header to gzip and includes a X‑Moz‑Crawler custom header in some requests.

📋 robots.txt Compliance

According to Moz’s official documentation (https://moz.com/help/guides/moz‑crawler/robots‑txt), the Page Analyzer bot fully respects Disallow directives in robots.txt. It also supports Crawl‑Delay instructions and will pause between requests accordingly. However, Moz notes that the bot may ignore Allow rules if they conflict with a broader Disallow, following the original RFC. The bot caches robots.txt for up to 24 hours per domain.

🔍 Detection Indicators

The primary User‑Agent string is Mozbot (often seen as “Mozbot/1.0” or “Mozbot/2.0”). A secondary string, MozPageAnalyzer, appears in requests directed at Moz’s own tools. The bot also sets a From header containing “[email protected]” and a User‑Agent that includes “Moz”. Behavioral fingerprints include a low request rate (max 10 requests per minute per domain) and a consistent pattern of fetching only HTML pages, never resources like PDFs or images.

📊 Data Usage

Collected data is used exclusively for Moz’s SEO analysis products. Page titles, meta tags, heading hierarchies, and internal link counts are stored in Moz’s index to generate site audit reports, page optimization scores, and competitive analysis dashboards. Moz does not use this data for AI training or public search indexing. The data is retained for up to 90 days for active campaigns, after which it is anonymized.

⚙️ Rate Limiting Policy

Although Page Analyzer is a legitimate SEO crawler, its aggregate crawl volume across tens of thousands of sites can strain smaller servers. Rate‑limiting is recommended at 100 requests per hour per IP, with a threshold‑based block (e.g., 503 status) applied if the bot exceeds that rate. The policy rationale is to prevent resource exhaustion while still allowing the bot to complete its analysis within reasonable timeframes, as Moz itself advises configuring Crawl‑Delay in robots.txt for fine‑grained control.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.