🤖 Overview

Checkbot is a legitimate SEO auditing web crawler operated by Checkbot Ltd., a UK-based company that provides the Checkbot.io platform for website diagnostics. First released in 2016 as a Chrome extension and later as a standalone SaaS tool, its primary purpose is to crawl websites to detect technical SEO issues, broken links, duplicate content, missing meta tags, and performance bottlenecks. The collected data feeds into the Checkbot dashboard, where users receive actionable reports to improve search engine visibility and site health. According to the official Checkbot documentation, the crawler is designed for individual webmasters, agencies, and enterprises who want automated site audits without manual inspection.

🌐 Technical Behavior

Checkbot performs both depth-first and breadth-first crawling depending on the configuration set by the user. By default, it crawls a site starting from a provided seed URL and follows internal links up to a configurable depth (typically 3–5 levels). The crawler respects the Maximum Crawl Delay directive in robots.txt and uses an adaptive delay that can be set as low as 100 milliseconds or as high as 10 seconds per request. Its request frequency is moderate, averaging 2–5 requests per second for a standard audit, but can be throttled further to avoid server strain. The IP ranges used by Checkbot are published on its official website and are all static, belonging to the Amazon Web Services (AWS) EC2 fleet, specifically in the 54.xxx.xxx.xxx and 52.xxx.xxx.xxx blocks. The crawler uses HTTP/1.1 with optional support for HTTP/2 and sends a standard User-Agent header (see Detection Indicators). It also respects the Cache-Control header to avoid cached responses when freshness is required.

📋 robots.txt Compliance

Checkbot fully honors the Robots Exclusion Protocol, including both Disallow directives and Crawl-Delay instructions. Based on its published technical documentation, the crawler reads the site’s robots.txt before each crawl session and will not access any URL listed under a Disallow rule. This compliance is verified through independent testing by webmaster communities and the official Checkbot blog. There are no recorded instances of Checkbot ignoring robots.txt rules, and the tool provides a dedicated warning if a site lacks a robots.txt file.

🔍 Detection Indicators

The primary User-Agent string for Checkbot is: Mozilla/5.0 (compatible; Checkbot/1.0; +https://checkbot.io/bot). Some older versions may appear as Checkbot/0.9 or Checkbot Crawler. Behavioral fingerprints include a high rate of sequential GET requests for HTML pages (rarely for images or CSS), and a consistent pattern of requesting robots.txt before any other resource. The crawler also sends a From header in some cases: [email protected]. It does not spoof its identity and always identifies itself in the User-Agent.

📊 Data Usage

Collected data is used exclusively for the user who initiated the crawl. Checkbot does not use the crawled content for any AI training, indexing, or public aggregation. Each report is private to the account that triggered the audit, and the raw data is stored for 30 days before automatic deletion. The platform’s privacy policy states that no personally identifiable information (PII) from the target site is retained after the report is generated, unless the user explicitly saves logs.

⚙️ Rate Limiting Policy

Checkbot is rate-limited by design because its adaptive delay mechanism can still generate bursts of up to 10 requests per second if misconfigured or if the site has minimal latency. Web application firewalls and server admins commonly impose a threshold of 20 requests per second over a 30-second window to protect against accidental overload while still allowing legitimate auditing. The rationale is to balance the utility of thorough SEO diagnostics with the server’s capacity, especially for smaller sites that may not expect high-frequency crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.