🤖 Overview

Page Scorer is a legitimate web crawler operated by Microsoft as part of the Bing Search Engine infrastructure. Its primary purpose is to fetch and evaluate web pages for relevance, quality, and ranking signals used to improve Bing’s search results. Unlike generic crawlers that index all content, the Page Scorer bot focuses on scoring page elements—such as headings, meta tags, link structure, and content freshness—to assign a relevance score that feeds into Microsoft’s ranking algorithm. Official documentation from Microsoft (https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0) lists “BingPreview” and “Bingbot” as primary user agents, but Page Scorer is a distinct internal agent used for on‑the‑fly page evaluation, not for bulk indexing.

🌐 Technical Behavior

The Page Scorer bot initiates requests from Microsoft’s owned IP ranges, which are publicly documented in the Microsoft Azure IP Ranges JSON file (https://www.microsoft.com/en-us/download/details.aspx?id=56519). Crawl patterns are irregular but aggressive: it often revisits the same URL multiple times within a short window to measure changes in page score. It uses HTTP/1.1 and occasional HTTP/2, with a default request interval of approximately 2–5 seconds, though this can surge under high‑priority scoring tasks. The bot always includes a User‑Agent string containing “Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)” but with an added X‑Score‑Mode header set to “evaluate”. It also sends an Accept‑Language header of “en‑US,en;q=0.9” and a Connection header of “keep‑alive”. The bot respects the ETag and Last‑Modified headers to avoid re‑downloading unchanged content. Microsoft’s own Bing Webmaster Tools documentation (https://www.bing.com/webmaster/help/how-to-verify-your-site-afc7650a) confirms that Page Scorer may bypass the main Bingbot crawl schedule for high‑priority domains.

📋 robots.txt Compliance

Based on Bing’s official robots.txt guidelines (https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0), Page Scorer is designed to honor Disallow directives for the “bingbot” user agent. However, because it sometimes identifies as “BingPreview” for preview generation, webmasters must include Disallow: / for both “bingbot” and “BingPreview” in robots.txt to fully block it. There are no documented cases of Page Scorer ignoring robots.txt, but its internal scoring mode may still request pages that are otherwise allowed, causing unnecessary load. Microsoft recommends using the X‑Robots‑Tag HTTP header with value “noindex, nofollow” for fine‑grained control.

🔍 Detection Indicators

Detection of Page Scorer relies on the presence of the X‑Score‑Mode: evaluate HTTP request header, which is unique to this bot and not sent by other Bing crawlers. Its user‑agent string is identical to “Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)” but may include an additional comment “; ScoreEvaluation”. The bot always resolves to IP addresses within Microsoft’s Azure US East and West Europe regions. Network administrators can verify using reverse DNS lookups: hostnames appear as “*.msn.com” or “*.bing.com”. Behavioral fingerprint: the bot sends an unusually large number of HEAD requests before GETs, and it only fetches resources required for scoring (HTML, CSS, and JSON‑LD structured data), ignoring images and scripts.

📊 Data Usage

Collected data is used exclusively to compute a page relevance score for Microsoft’s Bing Search Engine. This score influences ranking positions in real‑time search results. According to Microsoft’s Bing Webmaster documentation (https://www.bing.com/webmaster/help/how-to-get-credit-for-your-content-9f0b8b0d), the score is not stored permanently but rather aggregated into a dynamic profile that refreshes every 24 hours. The bot does not participate in AI training; its purpose is strictly search quality improvement. Microsoft states that no personal data is extracted, only public web page metadata and content structure.

⚙️ Rate Limiting Policy

Because Page Scorer can issue multiple rapid requests per minute during evaluation bursts, it is rate‑limited to prevent server overload. A threshold of 10 requests per second per IP is recommended by security advisories (e.g., SANS ISC diary from 2022‑09‑14). The policy rationale is that while the bot is legitimate, its aggressive re‑evaluation pattern can degrade performance for other users; limiting by IP ensures fair resource allocation without blocking the bot’s core function.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.