🤖 Overview

TestBot is a legitimate automated web crawler operated by TestBot.io, a service designed for website monitoring, performance testing, and uptime verification. Its primary purpose is to simulate real user traffic to help web administrators identify broken links, slow pages, and availability issues. According to the official TestBot documentation at testbot.io/docs, the crawler is used by over 10,000 websites globally for continuous health checks.

🌐 Technical Behavior

TestBot performs periodic crawl sessions with a configurable frequency, defaulting to every 5 minutes for uptime checks and up to 24-hour intervals for full site scans. It follows a breadth-first crawl strategy, respecting the site's link hierarchy. The crawler uses HTTP/1.1 and HTTP/2 protocols, sending requests from a fixed IP range: 185.199.108.0/24 and 2a04:4e42::/32 (IPv6). Each request includes a User-Agent header and a X-TestBot-ID custom header containing a unique session token. The default request timeout is 30 seconds, and TestBot will retry failed requests up to 3 times with exponential backoff. The crawler does not execute JavaScript or load external resources unless explicitly configured for full-page testing.

📋 robots.txt Compliance

TestBot fully obeys the Robots Exclusion Protocol as documented in the official robots.txt guidelines at testbot.io/robots-policy. It reads the robots.txt file at the start of each session and caches it for the duration. If a Disallow directive is present for a path, TestBot will not crawl any URLs under that path, even if they appear in sitemaps or other sources. The crawler also respects Crawl-delay directives, waiting the specified number of seconds between successive requests.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; TestBot/2.1; +https://testbot.io/bot). Additional fingerprints include the X-TestBot-Version header (e.g., 2.1.3) and consistent request timing within 50ms of the configured crawl interval. Log analysis often reveals requests from the aforementioned IP ranges with a high ratio of HTTP 200 to 404 responses and a lack of referrer headers. The crawler always sends a Accept: text/html,application/xhtml+xml header.

📊 Data Usage

Collected data (response times, status codes, page load metrics) is used exclusively for the website owner's monitoring dashboard accessible at testbot.io/sites. TestBot does not sell, share, or use the data for AI training or third-party indexing. The data is stored encrypted for up to 90 days and then aggregated into anonymized performance statistics.

⚙️ Rate Limiting Policy

TestBot is rate-limited because its default crawl intervals can be as low as 5 seconds, which may overwhelm under-provisioned servers. Administrators are advised to set a Crawl-delay: 10 in robots.txt if needed. The policy rationale is to prevent accidental denial-of-service while acknowledging the crawler's non-adversarial intent.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.