🤖 Overview

Cloudflare-AutoRAG is a legitimate web crawler operated by Cloudflare, Inc., introduced in May 2024 as part of the Cloudflare AutoRAG product. Its primary purpose is to index publicly accessible web content on behalf of Cloudflare customers, enabling retrieval-augmented generation (RAG) workflows for AI applications. The crawler is not malicious; it is a controlled, rate-limited agent designed to respect webmaster preferences.

🌐 Technical Behavior

The crawler originates from Cloudflare’s global anycast IP ranges, which are documented in Cloudflare’s published IP lists (e.g., 173.245.48.0/20, 103.21.244.0/22). It uses HTTP/2 and TLS 1.3 by default, mimicking modern browser behavior to avoid being blocked. Request frequency is moderate but can spike during initial indexing of a new domain; Cloudflare states the crawler will back off when it receives 429 responses. The bot respects Cache-Control headers and ETags to avoid re‑crawling unchanged content. It also obeys robots.txt directives and X-Robots-Tag headers, as confirmed in the official Cloudflare documentation.

📋 robots.txt Compliance

Cloudflare explicitly states that Cloudflare-AutoRAG honors robots.txt rules, including Disallow directives and Crawl‑Delay settings. Webmasters can block the bot entirely by adding a User-agent: Cloudflare-AutoRAG directive to their robots.txt file. This compliance is verified by Cloudflare’s own support articles and by community testing.

🔍 Detection Indicators

The bot identifies itself with the User‑Agent string Cloudflare-AutoRAG/1.0. It also sends a CF‑RAG‑Crawler header set to “1” for verification. Behavioral fingerprints include a consistent request pattern of fetching HTML and linked resources (CSS, JS, images) in rapid succession, but only on pages with visible text content. The IP addresses always belong to Cloudflare’s published ranges, which can be cross‑referenced with the official cloudflare.com/ips‑v4 list.

📊 Data Usage

Collected data is used exclusively to build knowledge bases for Cloudflare AutoRAG, a service that allows customers to create private RAG systems using their own website content. The crawled text is stored temporarily, processed into embeddings, and then served through Cloudflare’s Workers AI platform. No data is sold or shared with third parties; Cloudflare’s privacy policy governs all handling.

⚙️ Rate Limiting Policy

Cloudflare-AutoRAG is rate‑limited because it can send bursts of requests during initial indexing, potentially overwhelming smaller sites. A threshold‑based block (e.g., returning 429 Too Many Requests after 100 requests per minute) is a common, justified approach to protect server resources while still allowing the legitimate crawler to operate sustainably.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.