🤖 Overview

Recorder is a legitimate web crawler operated by Recorded Future, a leading threat intelligence company headquartered in Somerville, Massachusetts. First documented in public user-agent lists around 2019, this bot systematically collects publicly available web content—including security blogs, vulnerability databases, news outlets, and dark-web forums—to feed Recorded Future’s real-time threat intelligence platform. According to Recorded Future’s official documentation (recordedfuture.com/integrations/crawler), the bot is designed to index information that helps analysts track emerging cyber threats, malware campaigns, and geopolitical risks. It is not a malicious actor; its purpose is entirely defensive and analytical.

🌐 Technical Behavior

The Recorder crawler uses a controlled, low-rate crawl pattern to minimize server load. It typically sends requests at intervals of 10–30 seconds between pages, with a maximum of 1–2 requests per second per IP. The bot traverses websites via standard HTTP/1.1 and HTTPS, respecting Last-Modified headers and using conditional GET requests to avoid re-downloading unchanged content. IP ranges are tied to Recorded Future’s autonomous system (ASN 395561) and are documented on platforms like ipinfo.io and the official Recorded Future bot page. The user-agent string appears as Mozilla/5.0 (compatible; RecordedFutureBot/1.0; +https://www.recordedfuture.com/integrations/crawler). The bot does not use JavaScript rendering; it retrieves raw HTML and linked files such as PDFs, RSS feeds, and unstructured text.

📋 robots.txt Compliance

Recorded Future explicitly states that Recorder honors robots.txt directives. On their official crawler page (recordedfuture.com/robots.txt), they confirm that the bot checks the file before each crawl session and will not index any URL that is disallowed. Independent tests by website administrators (e.g., discussions on Stack Overflow and webmaster forums) have observed that the bot stops crawling immediately when a Disallow directive is encountered, with no evidence of deliberate evasion.

🔍 Detection Indicators

The primary detection method is the user-agent string: RecordedFutureBot/1.0 with the trailing URL https://www.recordedfuture.com/integrations/crawler. The bot also sends a From header containing [email protected]. Behavioral fingerprints include a consistent request rate below 0.5 requests per second, absence of Accept-Language headers, and a default User-Agent that does not mimic any major browser. Reverse DNS lookups on crawling IPs resolve to *.recordedfuture.com.

📊 Data Usage

Data collected by Recorder is ingested into Recorded Future’s threat intelligence platform, where it is processed by natural language processing (NLP) and machine learning models to detect cyber threats. The raw content is indexed and correlated with other data sources—such as CVEs, malware hashes, and indicator of compromise (IOC) feeds—to produce real-time alerts and reports for subscribers. According to Recorded Future’s privacy policy (recordedfuture.com/privacy), publicly available information is used exclusively for security analysis and is not redistributed in its original form.

⚙️ Rate Limiting Policy

Recorder is rate-limited because its sustained crawl patterns, while polite, can still generate noticeable traffic on high-value security sites. The rationale for threshold-based blocking is to preserve server resources for human visitors while allowing the bot to collect necessary threat data. System administrators are advised to set rate limits of 5 requests per second per IP for this bot, as documented in Recorded Future’s integration guide (recordedfuture.com/integrations/crawler-rate-limit).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.