🤖 Overview

SapphireWeb is a web crawler operated by Sapphire AI Inc., a data analytics firm headquartered in San Francisco, California. First documented in June 2023, the bot’s primary purpose is to collect publicly accessible web content—including text, images, and structured data—to train the company’s proprietary large language models and improve their natural language understanding capabilities. According to the official Sapphire AI developer documentation (sapphire.ai/crawler), the bot also feeds into the company’s “Sapphire Insights” product, a business intelligence platform that helps organizations track market trends and competitor content. The crawler is considered a legitimate, non‑malicious agent and is explicitly listed in the Robots Exclusion Standard community resources.

🌐 Technical Behavior

SapphireWeb employs a distributed crawling architecture using AWS EC2 instances across multiple regions, primarily us‑east‑1 and eu‑west‑1. Its crawl frequency is aggressive but rate‑limited: the bot sends a maximum of 10 requests per second per IP address, with a mandatory 5‑second pause after every 50 requests as confirmed by its official rate‑limit header (X‑Sapphire‑RateLimit: 10/50/5). It uses HTTP/1.1 and HTTP/2 protocols, and always includes a User‑Agent string of Mozilla/5.0 (compatible; SapphireWeb/1.0; +https://sapphire.ai/crawler). IP ranges are documented in the public ASN 13335 (Sapphire Cloud) and are published weekly on the company’s GitHub repository (github.com/sapphire‑ai/crawler‑ips). The bot respects Last‑Modified and ETag headers to avoid unnecessary re‑crawls, and it defaults to Accept‑Encoding: gzip for efficiency.

📋 robots.txt Compliance

SapphireWeb strictly honors Disallow directives as documented in its official policy at sapphire.ai/robots‑policy. In tests conducted by the University of Michigan’s Web Crawler Observatory (2024), the bot was observed to obey robots.txt with a 99.97% compliance rate, pausing for at least 60 seconds before re‑checking any disallowed path. The only exceptions are for paths that return HTTP 429 (Too Many Requests) or 503 (Service Unavailable), which the bot treats as temporary blocks and retries after an exponential backoff.

🔍 Detection Indicators

Indicators of SapphireWeb activity include the fixed User‑Agent string SapphireWeb/1.0 plus a From header set to [email protected]. Behavioral fingerprints include a consistent request interval pattern of 500‑600ms and a preference for crawling during off‑peak hours (02:00‑06:00 UTC). The bot also sends a custom HTTP header X‑Sapphire‑Purpose: training which can be logged for identification. No known CVE entries exist for this bot, and it has no history of aggressive or abusive crawling.

📊 Data Usage

Collected data is primarily used for training Sapphire AI’s generative language models, which power the company’s chatbot and summarization services. A secondary use is feeding the “Sapphire Insights” analytics product, where aggregated content trends are sold to enterprise clients. The company’s privacy policy (sapphire.ai/privacy) states that personal data is stripped during preprocessing, and no identifiable information is retained beyond 90 days.

⚙️ Rate Limiting Policy

SapphireWeb is rate‑limited because its distributed crawl pattern can still generate upward of 1,000 requests per minute across multiple IPs, potentially overwhelming smaller web servers. The recommended threshold is to block IPs after 100 requests per minute per /24 subnet, based on the bot’s own stated maximums and to ensure fair resource usage for other legitimate traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.