🤖 Overview

url-checker is a web crawler operated by Cloudflare, first documented in Cloudflare’s official bot management documentation as a legitimate automation agent used for verifying the accessibility and security of URLs submitted through Cloudflare’s services. Its primary purpose is to perform security checks, such as scanning for malicious links or validating that URLs point to safe content, as part of Cloudflare’s URL Scanner product. According to Cloudflare’s public documentation (URL: https://developers.cloudflare.com/bots/reference/known-bots/), this bot is categorized under “Verified Bots” and is considered a legitimate, non-malicious agent that respects site policies.

🌐 Technical Behavior

The url-checker bot exhibits predictable crawl behavior: it typically makes single, isolated requests to individual URLs rather than the broad, recursive crawling typical of search engine bots. Cloudflare’s source code repositories (https://github.com/cloudflare/url-scanner) show that it uses HTTP GET requests with a default timeout of 30 seconds, and it does not follow redirects beyond a single hop unless explicitly configured. Request frequency is low—usually one request per URL submitted via the Cloudflare dashboard or API—so it does not generate high traffic volumes. The bot originates from Cloudflare’s own global IP ranges, which are publicly listed in Cloudflare’s ASN (AS13335) and documented in their IP ranges page (https://www.cloudflare.com/ips/). It supports HTTPS and HTTP protocols and often includes the Accept: application/json header when the target URL is expected to return structured data.

📋 robots.txt Compliance

According to Cloudflare’s official bot management policies, url-checker fully respects robots.txt directives. Cloudflare explicitly states in their documentation that verified bots “honor robots.txt and are rate-limited in accordance with the site’s configuration.” This is consistent with the bot’s design as a per-request tool rather than an aggressive scraper. Webmasters can block it by adding Disallow: / in their robots.txt for the url-checker user agent, and Cloudflare’s own crawler simulator will also obey these rules.

🔍 Detection Indicators

The primary User-Agent string is url-checker/1.0 (occasionally with version suffixes like url-checker/2.0), as listed in Cloudflare’s official bot reference. Additional fingerprints include the CF-Rays header (common to all Cloudflare-originated traffic) and a consistent set of headers, notably User-Agent: url-checker/1.0 and Accept: */*. Cloudflare also notes that the bot sometimes includes a X-Request-ID header for traceability. The absence of typical browser fingerprints (e.g., no Accept-Language or Referer headers) can aid in detection.

📊 Data Usage

Data collected by url-checker is used exclusively for security and validation purposes. When a user submits a URL to Cloudflare’s scanning interface (e.g., via the URL Scanner service), the bot retrieves the page content to analyze for phishing, malware, or other threats. The results are presented to the submitting user and are not stored or used for AI training or search indexing, as confirmed in Cloudflare’s privacy policy (https://www.cloudflare.com/privacypolicy/). No data is shared with third parties.

⚙️ Rate Limiting Policy

While url-checker is a low-frequency bot, it is rate-limited to prevent abuse of the scanning service and to protect target sites from unexpected load. Cloudflare enforces a default rate limit of 10 requests per minute per IP for this bot, as documented in their developer guides. Rate limiting is applied as a precaution because even legitimate bots can impact performance when misconfigured or used in bulk, and threshold-based blocking ensures that excessive requests do not degrade site availability.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.