🤖 Overview

WRTNBot is a web crawler operated by WRTN, a data analytics company based in the United States that specializes in web content extraction and competitive intelligence. First publicly documented in 2020, its purpose is to systematically collect publicly available web data to feed into WRTN’s proprietary analytics platform, which provides insights into website changes, SEO performance, and content trends. The bot is classified as a legitimate, non‑malicious agent that adheres to industry standards and respects website owner preferences.

🌐 Technical Behavior

WRTNBot employs a distributed crawling architecture, issuing requests from a range of IP addresses primarily allocated to WRTN’s cloud infrastructure on Amazon Web Services (AWS) and Google Cloud Platform. Typical crawl frequency varies between 1 and 10 requests per second per domain, with longer delays automatically applied to larger sites. The bot uses HTTP/1.1 and HTTPS protocols and includes a custom HTTP header X‑WRTN‑Crawl: 1 for identification. Crawl patterns follow a breadth‑first strategy, starting from a sitemap or homepage, and may revisit pages based on a configurable freshness interval documented in WRTN’s technical white paper (https://wrtn.com/crawler‑architecture).

📋 robots.txt Compliance

According to WRTN’s official robots.txt policy page (https://wrtn.com/robots.txt‑policy), WRTNBot fully honors Disallow directives and respects custom Crawl‑delay values up to 60 seconds. The bot will not access any path marked Disallow and is reported to re‑evaluate robots.txt regularly during long crawls to incorporate any changes made by site administrators.

🔍 Detection Indicators

The primary User‑Agent string is WRTNBot/1.0 (compatible; WRTN; https://wrtn.com/bot) with version number increments. Additional identifying headers include From: [email protected] and the previously mentioned X‑WRTN‑Crawl: 1. The bot does not spoof other User‑Agents and can be verified via reverse DNS lookups on its IP ranges, which resolve to *.wrtn.com.

📊 Data Usage

Collected data is used exclusively for WRTN’s analytics services, which include monitoring website changes, tracking competitor content updates, and generating SEO reports for its customers. Data is not used for AI model training or sold to third parties; it remains within WRTN’s platform for aggregated customer insights, as stated in the WRTN privacy policy (https://wrtn.com/privacy).

⚙️ Rate Limiting Policy

WRTNBot is rate‑limited because its aggressive crawling patterns, if unrestricted, could overwhelm smaller servers or affect website performance for other users. A threshold‑based blocking policy—such as allowing up to 1,000 requests per minute per IP before applying a temporary ban—is recommended to balance the bot’s data collection needs with web server stability and fair resource allocation.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.