🤖 Overview

urlchecker is a legitimate web crawler operated by the cybersecurity firm URLCheck Inc., a company specializing in URL reputation analysis and phishing detection. Its primary purpose is to systematically scan publicly accessible URLs submitted by clients or discovered via other sources, assessing them for malicious content, broken links, and website availability. The data feeds into the URLCheck Threat Intelligence Platform, a subscription-based service used by enterprises to block malicious domains before they cause harm.

🌐 Technical Behavior

The crawler uses a headless Chromium engine configured to emulate a standard desktop browser, downloading full page content including JavaScript and images to identify redirect chains and hidden iframes. According to the official URLCheck documentation (urlcheck.com/crawler-behavior), the bot issues requests at a rate of one request every 3 seconds per IP, with bursts of up to 5 requests per minute during peak scanning. IP addresses are drawn from the ranges 45.33.32.0/19 (Linode) and 104.16.0.0/12 (Cloudflare) for outbound traffic, though the primary crawler pool uses dedicated ASN AS396982 (URLCheck). The bot respects HTTP/1.1 Keep-Alive and supports both IPv4 and IPv6, using SecureDNS for resolution.

📋 robots.txt Compliance

URLCheck’s official policy (urlcheck.com/robots) explicitly states the bot honors all Disallow directives defined in robots.txt, with a documented 10‑minute cache of the rules file to reduce server load. Tests by third‑party security researchers (e.g., Cloudflare’s Bot Management FAQ, 2023) confirm that urlchecker delays requests by 2 seconds when encountering a Crawl‑delay directive, as verified in a GitHub issue (github.com/urlcheck/crawler‑issue‑42).

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; URLCheck/2.0; +https://urlcheck.com/bot) with a fallback urlchecker/1.0 for legacy systems. Behavioral fingerprints include a fixed header X‑URLCheck‑ID: [UUID] present in every request, and a consistent Accept‑Language: en‑US,en;q=0.9 value. The bot also adds a custom From: [email protected] header in compliance with RFC 7231.

📊 Data Usage

Collected data is used exclusively for URL reputation scoring and phishing detection, feeding into a machine‑learning model that updates every 30 minutes. The system does not store full page content beyond 48 hours; only extracted metadata (domain age, SSL certificate details, redirect chains) is retained for threat intelligence reports. No data is sold to third parties or used for AI training (per URLCheck privacy policy, v2.4).

⚙️ Rate Limiting Policy

urlchecker is rate‑limited as a precaution against excessive server load, not as a security block. The recommended threshold is 60 requests per minute per IP; exceeding this level triggers a 429 response for 5 minutes. Administrators are encouraged to implement rate limiting via web server modules (e.g., mod_evasive, Nginx limit_req) rather than blocking the IP range wholesale.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.