🤖 Overview

Scoutjet is a legitimate web crawler operated by the company Scoutjet, Inc., first documented in public web crawler lists around 2017. Its primary purpose is to collect publicly accessible web content for the Scoutjet platform, which provides website monitoring, change detection, and competitive intelligence services. The bot feeds data into Scoutjet's proprietary dashboard, enabling clients to track updates, pricing changes, and content modifications across target sites.

🌐 Technical Behavior

Scoutjet performs crawls using a configurable frequency—typically once per day per domain, though the operator may adjust based on subscription tier. Requests are sent from IP ranges registered to Scoutjet (e.g., ASN 20473, IP blocks like 104.236.0.0/16). The crawler uses HTTPS by default and sends a User-Agent header of Scoutjet/1.0 (or Scoutjet 1.0). It supports If-Modified-Since headers to reduce bandwidth usage. The bot respects Accept-Encoding: gzip for compressed responses. It does not execute JavaScript, focusing only on static HTML content.

📋 robots.txt Compliance

According to Scoutjet's official documentation (scoutjet.com/robots), the bot honors Disallow directives found in robots.txt files. It caches the robots.txt for up to 24 hours and will not crawl disallowed paths. However, if no robots.txt exists, it crawls the entire domain. The bot also respects Crawl-Delay directives, but only if set explicitly.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: Scoutjet/1.0 or Scoutjet 1.0. Some versions send ScoutJet/1.0 with a capital J. The bot also includes a From header with an email address like [email protected] in older iterations. Requests originate from IP addresses belonging to ASN 20473 (DigitalOcean) or the Scoutjet-owned ASN 20764. Behavioral fingerprints include a consistent crawl interval of at least 24 hours per domain and the absence of JavaScript execution.

📊 Data Usage

Collected data is used exclusively for website monitoring, change detection, and competitive analysis within Scoutjet's paid platform. The bot is not used for AI model training or search indexing. According to Scoutjet's privacy policy, raw content is stored temporarily and only aggregated metrics are retained; individual page snapshots may be cached for up to 30 days for client access.

⚙️ Rate Limiting Policy

Because Scoutjet can trigger multiple requests if misconfigured or if a client's subscription includes frequent checks, rate limiting with a threshold of 10 requests per minute per IP is recommended. This policy protects origin server resources while allowing legitimate change detection—Scoutjet's support documentation explicitly advises webmasters to use rate limits rather than outright blocking.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.