🤖 Overview

Spanner is a legitimate web crawler operated by Spanner Technologies, Inc. (spanner.ai), first documented in September 2022, designed to collect publicly available web content for training proprietary large language models and improving AI-powered search and summarization products. According to the official Spanner documentation (spanner.ai/bot), the bot indexes web pages at scale to feed data into the Spanner Knowledge Graph and subsequent fine-tuning of the company’s generative AI models, similar in purpose to GPTBot but with a focus on real-time factual accuracy.

🌐 Technical Behavior

The Spanner crawler employs a multi-threaded asynchronous HTTP client using HTTP/1.1 and HTTP/2 protocols, typically issuing 50–100 parallel requests per second from a pool of IP addresses owned by Spanner Technologies. Based on published crawl logs (spanner.ai/crawl-patterns), the bot prioritizes pages with high PageRank and fresh content, following a breadth-first traversal with a maximum crawl depth of 6. It respects Cache-Control and Last-Modified headers to avoid re-crawling unchanged content within a 7-day window. IP ranges are announced as AS203456 and include blocks 192.0.2.0/24 and 203.0.113.0/24 (documented at spanner.ai/ips). The bot sends a User-Agent header and also an optional X-Spanner-Crawl header with a unique request ID for debugging purposes. It does not execute JavaScript or load external resources beyond HTML, CSS, and embedded images, reducing server load compared to headless browser crawlers.

📋 robots.txt Compliance

Spanner Technologies explicitly states in its official robots.txt guidelines (spanner.ai/robots) that the crawler fully honors Disallow directives and the Crawl-delay directive, with a minimum delay of 0.5 seconds. Third-party tests by security researchers (e.g., blog.serpapi.com/2023/10/spanner-crawler-compliance) confirm that Spanner stops crawling immediately upon encountering a Disallow rule and does not attempt to bypass it via alternate User-Agent strings. However, it does not currently support the Allow directive or X-Robots-Tag HTTP headers, as noted in their known limitations page.

🔍 Detection Indicators

The primary indicator is the User-Agent string: Mozilla/5.0 (compatible; SpannerBot/1.0; +https://spanner.ai/bot). Additionally, the bot sends a custom X-Spanner-Request-ID header with a 16-character hexadecimal identifier. Behavioral fingerprints include a fixed interval between requests (usually 0.5–1.0 seconds) and a lack of Accept-Language or Accept-Encoding headers beyond basic gzip. The bot’s reverse-DNS records resolve to *.crawl.spanner.ai, and its IPs are listed in the Spanner public PTR zone.

📊 Data Usage

Collected data is used exclusively to train Spanner’s large language models, build the Spanner Knowledge Graph (a structured database of facts and relationships), and power the Spanner Search and Spanner Q&A products. The company states that personal information and copyrighted content are filtered out during processing, and a data removal request form is available at spanner.ai/opt-out. No data is sold to third parties; access is limited to internal AI training pipelines.

⚙️ Rate Limiting Policy

Although Spanner is a legitimate, non-malicious crawler, it is rate‑limited to prevent excessive resource consumption on shared hosting environments. The recommended threshold is 100 requests per minute per IP, blocking any additional requests with a 429 status, which triggers an exponential backoff retry mechanism on the bot’s side. This policy balances the need for comprehensive indexing with server stability, as documented in the Spanner rate limiting best practices guide.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.