🤖 Overview

urlck is a legitimate web crawler operated by the company behind the SEO analytics platform UrlCK, which provides website auditing, broken link detection, and page performance monitoring. According to the official UrlCK documentation (urlck.com), the bot is designed to systematically traverse websites to collect metadata on hyperlinks, response codes, and page load times, feeding this data into the company’s client dashboard for SEO professionals and webmasters. It is not associated with any malicious activity and is categorized as a rate-limited automated agent.

🌐 Technical Behavior

The urlck crawler sends HTTP GET requests with a default interval of 2–5 seconds between pages, though it can adjust based on the crawl-delay directive if specified. It uses IPv4 addresses primarily from the AWS EC2 range (e.g., 52.xx.xx.xx and 54.xx.xx.xx) and may fall back to IPv6 if the target supports AAAA records. The bot identifies itself via the User-Agent string Mozilla/5.0 (compatible; urlck/1.0; +https://urlck.com/bot) and includes a From header with an optional contact email. It does not execute JavaScript or CSS, instead parsing raw HTML to extract anchor tags and other link elements. Requests are made over HTTPS by default, and the bot respects HTTP status codes (e.g., 403 or 429) to back off.

📋 robots.txt Compliance

Based on the official UrlCK governance page (urlck.com/robots-policy), urlck fully honors Robots Exclusion Protocol directives, including Disallow, Allow, and Crawl-Delay. The bot reads the robots.txt file before each crawl session and caches it for up to 24 hours. Evidence from community reports and webmaster forums confirms that urlck does not ignore explicit disallow instructions, though it may recheck the file periodically to detect changes.

🔍 Detection Indicators

The primary detection method is the User-Agent string urlck/1.0 or variation urlck-bot. The bot also sends a custom header X-Urlck-Crawler: true in every request for transparency. Log analysis reveals that urlck requests often include a Referer header set to https://urlck.com or the target URL itself. The bot does not spoof other User-Agents and consistently uses the same signature across visits.

📊 Data Usage

Collected data—such as link status codes, redirect chains, and page response times—is aggregated into the UrlCK platform for SEO auditing, broken link reporting, and performance benchmarking. According to UrlCK’s privacy policy (urlck.com/privacy), this dataset is not used for AI training or sold to third parties; it solely serves the platform’s analytics features. Webmasters can view crawl reports and use the data to improve site health.

⚙️ Rate Limiting Policy

Because urlck can send batches of hundreds of requests during a deep link audit, it is rate-limited to prevent server overload. The recommended threshold for web application firewalls is 20 requests per second per IP, with a temporary block (5 minutes) if exceeded, as advised by UrlCK’s operational guidelines for high-traffic sites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.