🤖 Overview

SiteOrbiter is a legitimate web crawling service operated by the company SiteOrbiter Ltd., primarily known as a cloud-based SEO audit and website analysis platform. Its purpose is to systematically crawl websites to generate detailed reports on page structure, broken links, metadata, load times, and other on-page SEO factors, helping webmasters identify technical issues. The data is fed into the SiteOrbiter dashboard, which provides visualizations and actionable recommendations for site optimization.

🌐 Technical Behavior

SiteOrbiter crawls using a configurable depth and pace, typically starting from a given seed URL and following internal links. According to its official documentation, it employs a polite crawling algorithm with a default delay of 0.5 seconds between requests to avoid overwhelming servers, though users can adjust the crawl speed via the control panel. The bot uses HTTP/1.1 and supports both IPv4 and IPv6 addresses; its outbound IP ranges are dynamically allocated from cloud providers (e.g., AWS, GCP) but can be obtained by requesting the SiteOrbiter IP ranges list from support. Requests are sent with a standard Accept-Encoding: gzip header and a User-Agent string that includes the crawl ID for traceability.

📋 robots.txt Compliance

SiteOrbiter honors robots.txt directives by default. Its crawler reads the Disallow rules and will not request those paths, as confirmed by the official Robots.txt and Crawl Behavior support article. However, the bot respects the Crawl-delay directive only if configured in the user’s crawl settings; otherwise, it uses the default delay. No documented evidence was found of the bot ignoring or bypassing robots.txt rules.

🔍 Detection Indicators

The primary User-Agent string for the crawl bot is Mozilla/5.0 (compatible; SiteOrbiter/2.0; +https://siteorbiter.com/bot). Additional identifying headers include a From field containing the registered user’s email (only if enabled) and a custom X-Crawl-ID header that matches the job identifier in the SiteOrbiter user dashboard. The bot also sends a Referer header set to the origin URL being crawled. Behavioral fingerprints include a consistently low request rate and a strict following of same‑origin links only (no external jump unless explicitly configured).

📊 Data Usage

Collected data is used exclusively for SEO and web performance analysis. SiteOrbiter does not train AI models or sell raw data to third parties; instead, it aggregates crawl results into structured reports that show technical SEO metrics, broken links, duplicate content, and page speed. The company states in its privacy policy that crawl data is stored only for the duration of the subscription and can be deleted upon request.

⚙️ Rate Limiting Policy

Although SiteOrbiter is a legitimate and polite crawler, it may generate high volumes of requests if misconfigured (e.g., low crawl delay set by the user). Rate limiting with a sensible threshold (e.g., 100 requests per minute from a single IP) is recommended to prevent accidental resource exhaustion and to maintain server stability for other traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.