🤖 Overview

The cfnetwork crawler is operated by Cloudflare, Inc., a major content delivery network and web security provider, and is used to perform automated network health checks, security vulnerability scanning, and performance monitoring across websites that use Cloudflare's services. According to Cloudflare's official documentation, cfnetwork is part of the Cloudflare Security Scanner and Health Check systems, which run periodic scans to identify misconfigurations, open ports, and potential security risks on customer origins.

🌐 Technical Behavior

cfnetwork conducts scans using TCP and HTTP/HTTPS probing, typically originating from Cloudflare's published IP ranges (e.g., 173.245.48.0/20, 103.21.244.0/22) and includes both IPv4 and IPv6 addresses. The crawler sends requests with a configurable frequency—often every 30 seconds to 5 minutes for health checks, and daily or weekly for security scans—and it respects HTTP status codes like 429 (Too Many Requests) as throttling signals. Cloudflare's official IP ranges list (available at https://www.cloudflare.com/ips/) shows that these scans originate from Cloudflare's own infrastructure, not from arbitrary user agents. The bot supports both HTTP/1.1 and HTTP/2 protocols and may include a CF-Connecting-IP header matching the origin IP.

📋 robots.txt Compliance

Cloudflare explicitly states that cfnetwork honors robots.txt directives when performing scans, as documented in their Security Scanner Best Practices guide. The bot reads the Disallow rules before crawling, and administrators can block specific paths by adding User-agent: cfnetwork entries (though Cloudflare recommends not blocking these scans because they help identify real vulnerabilities). In practice, the bot also respects Crawl-Delay directives where present.

🔍 Detection Indicators

The primary User-Agent string is cfnetwork (sometimes appearing as cfnetwork/1.0 or Cloudflare-Health-Check), and requests often include a CF-Ray header containing a Cloudflare data center code. Behavioral fingerprints include a consistent source IP from Cloudflare's published ranges, a User-Agent: cfnetwork header, and a Via header showing Cloudflare's proxy. The bot also sends requests with a Accept: */* header and no referrer.

📊 Data Usage

Collected data—such as response times, HTTP status codes, TLS certificate validity, and presence of vulnerable headers—is used exclusively by Cloudflare to generate Security Center reports and Health Check dashboards for website owners. This data is not used for AI training or resold; it remains within the customer's Cloudflare account to improve their security posture and uptime monitoring.

⚙️ Rate Limiting Policy

While cfnetwork is a legitimate agent, it is rate-limited because its aggressive scanning cadence (e.g., every 30 seconds) can overwhelm under-resourced origin servers or interfere with normal traffic. Threshold-based blocking (e.g., >100 requests per minute from a single Cloudflare IP) is a prudent policy to prevent unintended denial-of-service while still allowing the beneficial scans to proceed at a controlled rate.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.