🤖 Overview

PageAnalyzer is a legitimate web crawler operated by PageAnalyzer LLC, a company providing website performance and SEO analysis services. The bot systematically crawls publicly accessible web pages to collect technical data such as page load times, meta tags, heading structure, and broken link detection, which feeds into their cloud-based analytics platform for customers.

🌐 Technical Behavior

According to the official documentation at https://www.pageanalyzer.com/bot, the crawler uses a polite crawl rate with a default delay of 2–3 seconds between requests, configurable via the Crawl-Delay directive in robots.txt. It operates over HTTP/1.1 with persistent connections and supports gzip decompression to minimize bandwidth. The bot’s IP ranges are published in a dedicated IP list at https://www.pageanalyzer.com/ips.txt and belong to the ASN AS394408. The crawler only fetches HTML pages and avoids binary resources like images, CSS, or JavaScript unless explicitly configured by the customer. It uses HTTP ETags and If-Modified-Since headers to avoid re-crawling unchanged content, reducing server load.

📋 robots.txt Compliance

PageAnalyzer fully honors standard robots.txt directives, including Disallow and Crawl-Delay, as stated on their bot policy page at https://www.pageanalyzer.com/bot-policy. They also provide a contact email for site owners who wish to request additional restrictions or report issues.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; PageAnalyzer/1.0; +http://www.pageanalyzer.com/bot). Additional detection signals include a custom HTTP header X-PageAnalyzer-Bot: 1 and reverse DNS records resolving to *.pageanalyzer.com. The bot consistently sends a Accept-Language: en-US,en;q=0.9 header and does not store or transmit cookies.

📊 Data Usage

Collected data is used exclusively to generate website analysis reports for PageAnalyzer customers, covering metrics like page speed, SEO compliance, content quality, and link integrity. According to their privacy policy at https://www.pageanalyzer.com/privacy, no personal user data is harvested, and the data is not used for AI training, advertising, or any purpose beyond the customer dashboard.

⚙️ Rate Limiting Policy

PageAnalyzer is rate-limited because its crawls can become aggressive when multiple customers target the same site simultaneously. Most security tools recommend threshold-based blocking at 10 requests per second without a crawl delay, as the bot itself adheres to a default 2-second delay but may accelerate under customer configuration. This ensures server stability without falsely blocking legitimate analysis traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.