🤖 Overview

SiteAuditBot is a legitimate web crawler operated by Mangools, a company known for its SEO tool suite including the Site Audit feature. The bot is explicitly used to crawl websites on demand or on a scheduled basis for the purpose of generating SEO health reports, identifying technical issues such as broken links, duplicate content, missing meta tags, and slow-loading pages. According to Mangools’ official documentation available at https://www.mangools.com/site-audit-bot, the crawler only visits URLs that users explicitly submit through the Mangools Site Audit tool and does not perform web-wide discovery or index the entire internet. It is a task-specific agent, not a general-purpose search engine crawler.

🌐 Technical Behavior

The SiteAuditBot uses a crawl pattern that mirrors a standard web browser, sending HTTP/1.1 requests with a configurable user-agent string that always includes “SiteAuditBot/1.0”. The default request rate is moderate and can be throttled by the user through the Mangools interface; by default, the bot sends approximately one request every 1–3 seconds per crawling session. The bot is designed to follow internal links recursively within the domain specified by the user, but it does not follow external outbound links. IP ranges are drawn from a dynamic pool of cloud-based proxy servers, primarily from Amazon Web Services (AWS) and DigitalOcean, making it difficult to block by IP alone. The bot supports both HTTP and HTTPS protocols, respects robots.txt crawl-delay directives, and sends Accept-Encoding: gzip headers to reduce bandwidth. It also respects noindex meta tags and nofollow attributes on links.

📋 robots.txt Compliance

Mangools explicitly states on their SiteAuditBot information page that the crawler fully honors robots.txt directives, including Disallow rules and Crawl-Delay values. The bot reads the robots.txt file at the start of each crawl session and will not access any URL that is disallowed. This behavior is verified by multiple third-party SEO bloggers and Mangools’ own support articles. No reports have been found of the bot ignoring robots.txt; it is considered one of the more compliant SEO audit bots in the industry.

🔍 Detection Indicators

The primary User-Agent string for SiteAuditBot is Mozilla/5.0 (compatible; SiteAuditBot/1.0; +http://www.mangools.com/site-audit-bot). Some older versions may have used a slightly different format, but the substring “SiteAuditBot” is always present. The bot does not send any other identifying headers besides standard browser-like headers. Behavioral fingerprints include a consistent request interval, a lack of JavaScript execution, and a narrow crawl depth limited to the starting domain. Access logs from web servers will show the user-agent string and typical IP ranges from AWS (e.g., 54.x.x.x, 52.x.x.x) or DigitalOcean (e.g., 138.x.x.x, 159.x.x.x).

📊 Data Usage

Data collected by SiteAuditBot is used exclusively for the Mangools Site Audit product to generate SEO reports for paying users. The bot extracts page titles, meta descriptions, headings, internal and external links, image alt text, page load times, HTTP status codes, and HTML structure. No content is used for AI training, search indexing, or any purpose beyond the user’s own audit dashboard. Mangools’ privacy policy confirms that crawl data is stored temporarily and is not shared with third parties. The data is only accessible to the user who initiated the audit.

⚙️ Rate Limiting Policy

Although SiteAuditBot is legitimate and well-behaved, web administrators often rate-limit it to protect server resources during peak traffic hours. Because the bot can be launched by multiple users simultaneously and may target the same domain from different accounts, threshold-based blocking (e.g., 10 requests per second from the same IP or user-agent pattern) is a reasonable precaution to prevent unintentional overload while still allowing scheduled audit crawls.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.