🤖 Overview

page_verifier is a legitimate crawler operated by Cloudflare, used to validate and verify the integrity of web pages as part of Cloudflare's security and content delivery network (CDN) infrastructure. It was first identified in early 2024 and is primarily used to check for content freshness, SSL certificate status, and routing correctness across Cloudflare’s global network. The bot's purpose is to ensure that pages served to end users are accurate and not cached versions that have been tampered with.

🌐 Technical Behavior

page_verifier crawls pages at a moderate frequency, typically sending a single request per URL per verification cycle, with intervals ranging from minutes to hours depending on the site's traffic and configuration. It uses HTTP/1.1 and HTTP/2 protocols and originates from Cloudflare's own IP ranges (e.g., 173.245.48.0/20, 103.21.244.0/22) as documented in Cloudflare's official IP ranges list. The bot does not follow redirects recursively; instead, it verifies the final response status and checks for specific security headers. It respects standard HTTP methods and does not send POST requests.

📋 robots.txt Compliance

According to Cloudflare's documentation, the page_verifier bot fully respects robots.txt directives. If a site's robots.txt file disallows access to certain paths, the bot will not crawl those pages. However, Cloudflare recommends using the Disallow directive to block this bot only if absolutely necessary, as bypassing verification could affect CDN performance tuning. The bot is listed in the robots.txt as User-agent: page_verifier and follows standard syntax.

🔍 Detection Indicators

The primary User-Agent string for this bot is "page_verifier" (without version numbers). It may also include a comment like "(Cloudflare)" in some instances. Additional headers include X-Forwarded-For matching Cloudflare’s IPs, and a CF-Ray header in some cases. The bot does not carry any cookies or session identifiers. Behavioral fingerprints include a low request rate (typically under 0.5 requests per second) and a lack of JavaScript rendering.

📊 Data Usage

The data collected by page_verifier is used exclusively by Cloudflare for internal monitoring and optimisation of its CDN and security services. Specifically, it verifies that the correct content is served, that SSL certificates are valid and not expiring, and that routing across Cloudflare’s edge nodes is efficient. No data is stored for AI training or sold to third parties; all collected information is ephemeral and aggregated for performance analysis.

⚙️ Rate Limiting Policy

page_verifier is rate-limited at the application layer to prevent abusive submission of verification requests. Threshold-based blocking is justified because the bot’s requests are essential for CDN health but could be used as a vector for denial-of-service if spoofed. Standard rate limiting (e.g., 10 requests per second per IP) is recommended to balance verification needs with server load.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.