🤖 Overview

SiteFinder is an automated web crawler operated by SiteFinder Inc. (sitefinder.com), a company specializing in website analytics, SEO auditing, and competitive intelligence. The crawler systematically indexes publicly accessible web pages to feed data into SiteFinder’s proprietary platform, which provides clients with insights on content structure, backlink profiles, and page performance metrics. First documented in 2018, the bot is designed for benign, legitimate website monitoring and is not associated with any malicious activities.

🌐 Technical Behavior

The SiteFinder crawler operates with a configurable request frequency that defaults to 1 request per second per domain, though users can adjust this via a dedicated control panel. It uses HTTP/1.1 and HTTP/2 protocols, sending requests with a randomized crawl delay of 500 ms to 2 seconds to reduce server load. IP ranges are drawn from a pool of 10,000 IPv4 addresses owned by SiteFinder Inc. and hosted on Amazon Web Services (us-east-1, eu-west-1) and Google Cloud (us-central1, europe-west4). The bot respects the Last-Modified and ETag headers to avoid re-crawling unchanged content. It also sends a User-Agent token and a reserved From header containing a contact email ([email protected]) per RFC 1945.

📋 robots.txt Compliance

According to SiteFinder’s official documentation (https://www.sitefinder.com/crawler/robots) the bot fully honors Disallow directives in robots.txt with a 10-second cache for rule files. It also supports the Crawl-Delay directive, throttling to a minimum of 5 seconds if specified. No instances of non-compliance have been reported in public server logs or security advisories.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; SiteFinder/2.0; +https://www.sitefinder.com/crawler). A secondary string used for mobile emulation is SiteFinder-mobile/1.0. Behavioral fingerprinting reveals consistent request patterns: the crawler always includes a Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 header and a Connection: keep-alive header. IPs can be identified via reverse DNS lookups resolving to *.sitefinder.com.

📊 Data Usage

Collected data — including page titles, meta descriptions, headings, links, and structured data — is used exclusively to populate SiteFinder’s dashboard for actionable SEO insights. The platform does not train generative AI models; instead, data is aggregated into anonymized reports for customers. Raw page content is stored temporarily (30 days) and then discarded according to the company’s privacy policy published at https://www.sitefinder.com/privacy.

⚙️ Rate Limiting Policy

SiteFinder’s default crawl rate of 1 request per second is intentionally conservative to minimize server impact, but site owners may apply threshold-based blocking (e.g., banning IPs that exceed 5 req/s) to protect resources. The rationale is that any site experiencing unexpected load from the bot should contact SiteFinder support to adjust the crawl window rather than permanently blocking the agent.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.