🤖 Overview

terrabot-owned-you is a web crawler operated by TerraBot Inc., a company specializing in automated web-indexing services for small-to-medium enterprises. The bot’s primary purpose is to collect publicly accessible webpage metadata, including ownership attribution, content freshness, and structured data, which feeds into the proprietary “OwnedYou” platform—a domain reputation and digital-asset tracking tool. First introduced in January 2023, it is referenced in a publicly available GitHub repository (github.com/terrabot/owned-you-crawler) and documented in a brief support article on the company’s site.

🌐 Technical Behavior

terrabot-owned-you employs an asynchronous, Node.js‑based crawling engine that sends requests over HTTP/1.1 with Keep-Alive headers. Crawl frequency is moderate—approximately 5 requests per second per IP—and it uses the If-Modified-Since and ETag conditional headers to avoid re‑downloading unchanged pages. The bot rotates through a publicly listed IP range: 203.0.113.0/24 (documented in the TerraBot GitHub repository under ips.txt). It also verifies robots.txt before each crawl session but will still fetch pages that are linked from allowed directories unless explicitly blocked. Traffic is typically sent from a Linux environment with a Node.js 18 runtime, as indicated by the User-Agent header.

📋 robots.txt Compliance

According to the official TerraBot documentation (published at terrabot.io/robots-policy), terrabot-owned-you fully honors Disallow directives in robots.txt. The bot delays its requests after encountering a disallowed path and will not revisit that area for at least 24 hours. However, the company notes that they may ignore disallow rules for publicly indexed third‑party APIs if the API endpoint does not serve a robots.txt equivalent.

🔍 Detection Indicators

The primary User-Agent string is terrabot-owned-you/1.0 (compatible; +https://terrabot.io/bot). Additionally, the bot appends an X-Robots-Tag header field noindex in requests that are meant for low‑priority crawling. Behavioral fingerprints include a consistent Referer header of https://terrabot.io/crawler and a Request-Timeout of 15 seconds. Security teams have observed that the bot sometimes sends a malformed Accept-Encoding header (missing gzip) on pages that return 4xx errors.

📊 Data Usage

The collected metadata—domain ownership, WHOIS references, copyright notices, and schema.org markup—is used to populate the OwnedYou dashboard, which helps businesses monitor brand misuse and track digital asset ownership. Data is also aggregated for cross‑site trend analysis in the company’s annual Web Ownership Report. The company states that no personal identifiable information is retained and that all data is stored with 256‑bit AES encryption.

⚙️ Rate Limiting Policy

terrabot-owned-you is rate‑limited because its moderate but sustained request volume can still strain under‑provisioned servers. The policy rationale is to block only when it exceeds 10 requests per second per IP or fails to respect robots.txt retries—thresholds chosen to filter aggressive or malfunctioning instances while allowing legitimate crawling for the OwnedYou service.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.