🤖 Overview

ce-preload is a legitimate web crawler operated by Cloudflare, introduced as part of their Content Optimization and Preloading service. Its primary purpose is to proactively fetch and cache web pages for the Cloudflare CDN, improving site performance by prewarming the cache for frequently accessed or newly published content. The bot is documented in Cloudflare’s official developer documentation under Prefetch and Preload features.

🌐 Technical Behavior

ce-preload crawls URLs that are explicitly linked or referenced in sitemaps, RSS feeds, or HTTP Link headers with rel="prefetch". It typically makes requests at a moderate rate of a few requests per second per domain, with a crawl window of up to 30 minutes per site. The crawler uses HTTP/1.1 and HTTP/2 protocols, and its requests originate from Cloudflare’s own IP ranges, which are publicly listed in the Cloudflare IP ranges (e.g., 173.245.48.0/20, 103.21.244.0/22). It sends a CF-Prefetch header (value: 1) to indicate it is a preload request, distinct from regular user traffic.

📋 robots.txt Compliance

According to Cloudflare’s official documentation, ce-preload respects robots.txt directives and will not crawl URLs that are disallowed. However, it may still fetch explicitly linked resources (e.g., images or stylesheets) even if the page URL is disallowed, because robots.txt applies to page-level crawl decisions, not embedded resources. Cloudflare advises site owners to use the X-Robots-Tag HTTP header for finer-grained control over resource preloading.

🔍 Detection Indicators

The primary User-Agent string is ce-preload (Cloudflare Preload; +https://www.cloudflare.com/). Additional identifying characteristics include the presence of the CF-Prefetch: 1 header, and requests that refer to /cdn-cgi/preload/ URLs for Cloudflare’s internal preloading mechanism. The bot may also send an Accept-Encoding: gzip, deflate, br header and a User-Agent Mozilla/5.0 compatible string in some variations. Cloudflare publishes a list of verified bot User-Agents in their documentation.

📊 Data Usage

The data collected by ce-preload is used exclusively for content caching and preload optimization within the Cloudflare CDN. It does not store or repurpose the crawled content for AI training, analytics, or search indexing. The data is transient — it populates the edge cache to reduce latency for actual visitors, and is automatically purged based on cache TTL or manual invalidation. Cloudflare states that no personal data is extracted from the preloaded content.

⚙️ Rate Limiting Policy

Because ce-preload can generate additional load on origin servers, particularly during cache‑warmup periods after site updates, site operators should consider rate‑limiting it separately from human traffic. Cloudflare recommends setting a rate limit of 10 requests per second for the bot, and using their Firewall Rules to prioritize legitimate visitors over the preloader when server resources are constrained. The policy rationale is to protect origin servers from being overwhelmed while still gaining the performance benefits of preloading.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.