🤖 Overview

cfetch is a legitimate web crawler operated by Cloudflare, Inc., documented in their official developer portal (Bot Management documentation). Its primary purpose is to fetch content from websites to support Cloudflare’s security features—such as Bot Management and Rate Limiting—as well as to warm CDN caches and validate SSL certificates. The crawler is not part of any AI training pipeline; instead, it feeds data into Cloudflare’s internal threat detection and performance optimization systems.

🌐 Technical Behavior

cfetch makes HTTP/1.1 and HTTP/2 requests from Cloudflare’s own IP address ranges, which are publicly listed in the Cloudflare IPs dataset (https://www.cloudflare.com/ips/). The crawler uses a variety of request methods, including GET and HEAD, and sends the User-Agent header "cfetch" (or "Cloudflare-cfetch" in some legacy versions). It typically does not follow redirects beyond a depth of 5, and its crawl frequency is moderate but can spike during cache warming operations. Cloudflare’s official guidance notes that cfetch respects Connection: close headers and does not support HTTP/3 for crawling. The crawler often requests /robots.txt first to check for disallowed paths before proceeding.

📋 robots.txt Compliance

cfetch fully honors robots.txt directives. Cloudflare’s developer blog and support documentation explicitly state that the crawler reads and follows Disallow rules (Cloudflare Bot Management Best Practices). There is no evidence that cfetch ignores or bypasses robots.txt, making it a well-behaved agent suitable for inclusion in a rate‑limited but not blocked access policy.

🔍 Detection Indicators

The primary identification is the User-Agent string cfetch or, occasionally, Mozilla/5.0 (compatible; Cloudflare-cfetch/1.0; +https://www.cloudflare.com/). Additional behavioral fingerprints include a very consistent request interval (often 1–5 seconds between pages), the absence of JavaScript rendering, and the fact that all requests originate from Cloudflare AS13335. Security teams can verify the IP against Cloudflare’s published IP lists to confirm authenticity.

📊 Data Usage

Data collected by cfetch is used exclusively for Cloudflare’s internal security and performance products. This includes analyzing traffic patterns to train Bot Management models, prefetching content to reduce origin server load, and verifying TLS certificate status. No data is used for AI language model training, search indexing, or third-party analytics. Cloudflare’s privacy policy further guarantees that crawled content is not stored longer than necessary for these operational purposes.

⚙️ Rate Limiting Policy

While cfetch is not malicious, it can produce high volumes of requests during cache warming or health-check cycles, which may cause performance degradation on smaller servers. A threshold‑based rate limit (e.g., 100 requests per minute per IP) is recommended to protect origin infrastructure while still allowing the legitimate crawler to perform its functions.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.