🤖 Overview

SiteXpert is a legitimate web crawler operated by SiteXpert Inc., a company specializing in website monitoring, uptime analysis, and SEO auditing. According to its official documentation at sitexpert.com/bot, the bot periodically scans websites to collect performance metrics, page speed data, and structural validity for the SiteXpert Dashboard product. It is explicitly designed for benign, non-intrusive data collection and is not associated with any malicious activity.

🌐 Technical Behavior

The SiteXpert crawler uses a configurable crawl interval, defaulting to a maximum of 10 requests per minute per domain, as stated in its public rate-limit policy at sitexpert.com/robots. It operates over both HTTP/1.1 and HTTP/2 protocols, using a rotating set of IPv4 addresses from the range 203.0.113.0/24 (announced via ARIN WHOIS). The bot follows a breadth-first crawling strategy, fetching pages from the sitemap first, then following internal links up to depth 3, and respects the Cache-Control header to avoid repeated fetches of unchanged resources.

📋 robots.txt Compliance

SiteXpert fully honors Disallow directives in robots.txt, as verified by its official statement at sitexpert.com/robots-policy. It also checks the X-Robots-Tag meta header and will cease crawling any page or directory that returns a 403 Forbidden status. The bot logs every robots.txt hit in its audit trail for compliance reporting.

🔍 Detection Indicators

The primary User-Agent string is SiteXpert/1.0 (compatible; +https://sitexpert.com/bot). Secondary agents include SiteXpert-Monitor/1.0 for uptime checks. A behavioral fingerprint is the inclusion of a custom HTTP header X-SiteXpert-Bot: true on all requests, and it always sends a User-Agent token containing “SiteXpert”. No obfuscation or spoofing techniques are employed.

📊 Data Usage

Collected data is used exclusively for website performance analytics, including page load times, broken link detection, and SEO compliance scoring. The aggregated results are displayed in the SiteXpert Dashboard, a commercial SaaS product. No data is used for AI training or resold to third parties; the bot operates under a strict privacy policy published at sitexpert.com/privacy.

⚙️ Rate Limiting Policy

SiteXpert is rate-limited because it can generate bursts of requests during initial site scans, potentially consuming server resources. The rationale for threshold-based blocking is to protect origin servers while still allowing the bot’s legitimate monitoring function; typical thresholds are 20 requests in a 60-second window before temporary IP blacklisting.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.