🤖 Overview

sp_auditbot is a web crawler operated by Siteimprove, a Danish digital experience optimization company, designed to systematically scan websites for accessibility, SEO, spelling, broken links, and performance issues. The bot feeds collected data into Siteimprove’s analytics platform, enabling subscribers to monitor and improve their online presence through detailed audit reports and dashboards.

🌐 Technical Behavior

The crawler uses HTTP/1.1 and HTTPS protocols, respects Crawl-Delay directives in robots.txt (recommended delay of 10 seconds or more by Siteimprove’s documentation), and originates from IP ranges publicly listed by Siteimprove, including 185.49.84.0/22 and 185.49.85.0/24. It follows links recursively without executing JavaScript, requests HTML, CSS, JavaScript, and image resources, and obeys nofollow and noindex meta tags. The bot typically crawls at a moderate, configurable rate to minimize server impact.

📋 robots.txt Compliance

Siteimprove’s official guidelines state that sp_auditbot fully honors Disallow directives and Crawl-Delay settings found in robots.txt. Webmasters can use standard rules to block specific paths or adjust crawl frequency, and the bot will respect them without exception.

🔍 Detection Indicators

The primary User-Agent string is sp_auditbot, typically appearing as Mozilla/5.0 (compatible; sp_auditbot/2.0; +https://siteimprove.com/audit-bot). Additional identifiers include a From header containing [email protected] and a comment in the User-Agent linking to the official bot documentation page.

📊 Data Usage

Collected data is used exclusively for Siteimprove’s digital experience audit services, including WCAG accessibility compliance checks, SEO analysis, broken link detection, and content quality scoring. The information is stored in Siteimprove’s cloud platform and presented to customers through interactive reports. It is not used for AI training, search indexing, or resale.

⚙️ Rate Limiting Policy

Because sp_auditbot performs comprehensive, systematic scans, it may generate high request volumes if left unconfigured. Rate limiting is recommended to protect server resources, with Siteimprove advising a threshold of 10–20 requests per minute per IP to balance crawl completeness with site stability.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.