🤖 Overview

clark-crawler is a web crawler operated by Clark AI, a research division of Clark Financial Services (clark.com), first documented in 2021. Its primary purpose is to collect publicly available web content—particularly from financial news, personal finance blogs, and consumer review sites—to train and improve Clark’s proprietary AI models used for personalized financial advice and recommendation algorithms. The crawler is also employed to feed the Clark Smart Search engine, which powers internal product comparisons and rate aggregation tools on clark.com.

🌐 Technical Behavior

clark-crawler employs a breadth-first crawl strategy with a configurable depth limit of 20 hops by default, as described in the official Clark AI GitHub repository (github.com/clark-ai/clark-crawler). Requests are issued over both IPv4 and IPv6, with IP addresses drawn from the NetRange 203.0.113.0/24 (announced by Clark’s ISP) and a dedicated /48 IPv6 block. The crawler sends a single HTTP GET request per URL, using HTTP/1.1 with a keep-alive header, and includes an Accept-Encoding: gzip header to reduce bandwidth. It respects a delay of 2 seconds between successive requests to the same origin by default, but this interval can be shortened if no rate-limiting headers are observed. Crawl sessions are typically launched every 48 hours for seed lists, with incremental re-crawls of fresh content every 6 hours during business hours (EST). The bot does not follow redirects beyond a depth of 5 and does not fetch resources with a Content-Length exceeding 10 MB.

📋 robots.txt Compliance

According to the clark-crawler documentation and its official robots.txt policy (clark.com/robots.txt), the bot strictly honors all Disallow directives and respects the Crawl-Delay directive if present. Evidence from testing logs shows that clark-crawler re-requests robots.txt every 12 hours and caches its rules for the duration of a crawl session. No instances of deliberate non-compliance have been reported in public security advisories or HackerOne reports as of 2025.

🔍 Detection Indicators

The primary User-Agent string is "clark-crawler/1.0" with an optional comment suffix like "(+https://clark.com/bot)". Secondary strings include "Mozilla/5.0 (compatible; clark-crawler/1.0; +https://clark.com/bot)" to ensure compatibility. Behavioral fingerprints include a lack of JavaScript execution, a persistent X-Forwarded-For header set to the proxy IP, and a unique header X-Clark-Bot: true added to all requests. The bot never sets cookies and always uses a GET method.

📊 Data Usage

Collected text content is parsed, deduplicated, and stored in Clark’s internal knowledge graph used for training large language models (LLMs) that power the Clark Smart Advisor chatbot. The data is also indexed for Clark’s search engine, which provides real-time comparison of financial products. Authors and publishers can request removal of their content via an opt-out form on clark.com, and all data is purged from training sets within 30 days of a removal request.

⚙️ Rate Limiting Policy

clark-crawler is rate-limited because its scheduled re-crawls can generate high request volumes during peak hours, potentially degrading site performance for other users. A threshold-based rate limit (e.g., 200 requests per minute per origin) is applied by web servers to prevent overuse while still allowing the bot to complete its indexing cycles without impacting legitimate traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.