🤖 Overview

PageScorer is a legitimate web crawler operated by PageScorer Inc., a SaaS company that provides an SEO page analysis platform. First publicly documented around 2021, the bot’s primary purpose is to collect publicly accessible web content and compute a proprietary “page score” that evaluates factors such as page speed, content quality, meta-tag completeness, mobile responsiveness, and backlink signals. The data feeds the PageScorer dashboard, where website owners and SEO professionals can monitor and improve their pages. According to the official documentation at https://pagescorer.com/bot, the crawler is designed to be transparent and respectful of webmaster rules.

🌐 Technical Behavior

PageScorer uses a distributed crawling architecture with a moderate request frequency of roughly one request every 3–5 seconds per domain by default, though it can be configured to a slower pace via the robots.txt Crawl-Delay directive. The bot supports both HTTP/1.1 and HTTP/2 protocols and sends a User-Agent header clearly identifying itself. Crawling occurs from a range of IPv4 addresses that are not published in a consolidated list, but the official site advises that the bot’s IPs belong primarily to US-based cloud providers such as AWS and DigitalOcean. The crawler follows internal redirect chains and respects nofollow attributes on links. It also caches page content for up to 7 days before re-checking for updates. No JavaScript rendering is performed; only static HTML is analyzed.

📋 robots.txt Compliance

Based on the vendor’s documentation and published reports (e.g., https://pagescorer.com/robots-txt-policy), PageScorer fully honors Disallow directives in robots.txt. It also respects per-URL directives such as Allow and Crawl-Delay. The bot does not make any attempt to circumvent exclusions, and it advertises its compliance as a core trust feature. Webmasters can also block the bot entirely by setting a Disallow: / rule for user-agent PageScorer.

🔍 Detection Indicators

The primary identifier is the User-Agent string: PageScorer/2.0 (+https://pagescorer.com/bot). A secondary variant PageScorer/1.0 may be encountered on older crawls. In addition to the User-Agent, the bot sends a custom HTTP header X-PageScorer-Crawl: 1 to help with identification. Log entries will show no other identifying cookies or special fingerprints. The crawler’s host appears in reverse DNS records under the domain crawler.pagescorer.com.

📊 Data Usage

The collected data is used exclusively inside the PageScorer subscription-based analytics dashboard. The service computes a score from 0–100 for every crawled page, based on over 75 scoring criteria including heading structure, image alt attributes, and load speed. No data is sold to third parties or used for AI model training. The company’s privacy policy explicitly states that the crawler only processes public pages and does not store personal data.

⚙️ Rate Limiting Policy

Because PageScorer can send a sustained stream of requests when analyzing large sites, it is rate-limited to prevent excessive load on origin servers. The recommended policy is to set a Crawl-Delay of 10 seconds in robots.txt or to block the bot entirely if its activity interferes with normal site operations. Threshold-based blocking is applied only when the bot’s volume exceeds reasonable levels (e.g., more than one request per second).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.