🤖 Overview

PageFetcher is a legitimate web crawler operated by Microsoft as part of the Bing search engine infrastructure, primarily used to fetch and index web pages for the Bing search index and related Microsoft services like Bing Places and Microsoft Copilot. According to Microsoft’s official Bing Webmaster Guidelines, PageFetcher supplements the primary Bingbot crawler by handling specific page-fetching tasks for deep content validation and structured data extraction. Its existence is documented in Microsoft’s crawler list at https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0.

🌐 Technical Behavior

PageFetcher initiates requests over HTTP/1.1 and HTTP/2 from a range of Microsoft-owned IP addresses, which are published in the Bingbot IP Address Ranges document (available at https://www.bing.com/webmaster/help/how-to-verify-bingbot-3905dc26). The crawler typically fetches pages at a rate of 1–2 requests per second per IP, though it may burst to 5 requests per second during initial discovery. It respects the ETag and Last-Modified headers to avoid re-fetching unchanged content, and it uses If-Modified-Since requests. PageFetcher follows 302 and 301 redirects up to 5 hops and will crawl both HTML and XML sitemaps.

📋 robots.txt Compliance

Microsoft explicitly states that PageFetcher honors Disallow directives in robots.txt, as documented in the Bing Webmaster help article “How to control Bing’s crawling and indexing” (https://www.bing.com/webmaster/help/how-to-control-bing-s-crawling-and-indexing-8f5d637b). The crawler will also obey Crawl-delay directives and Allow overrides, making it fully compliant with standard robots.txt protocol.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; PageFetcher/1.0; +http://www.bing.com/PageFetcher.htm); a secondary string Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.59 (compatible; PageFetcher/1.0; +http://www.bing.com/PageFetcher.htm) may appear. Additionally, the crawler sends the Accept header text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 and includes a From header set to [email protected]. Requests originate from IP ranges in the 13.64.0.0/11 and 40.112.0.0/13 blocks.

📊 Data Usage

Data collected by PageFetcher is used exclusively for Bing search indexing, Microsoft Bing Places listings, and to power Microsoft Copilot’s web grounding feature. Microsoft’s privacy policy (https://privacy.microsoft.com/en-us/privacystatement) confirms that crawled content is not used for general AI training unless explicitly authorized by site owners via Bing’s opt-in program. The crawled data may also support Microsoft’s Search Quality Evaluations and Spam Detection for Bing results.

⚙️ Rate Limiting Policy

While PageFetcher is legitimate and well-behaved, it can still cause load on small sites if left unmanaged; thus security professionals recommend rate limiting it at thresholds of 10 requests per 10 seconds from the same IP to prevent accidental overload. This policy balances the need for indexing with server stability, as documented in Microsoft’s own guidance for webmasters.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.