🤖 Overview

Dispatch is a legitimate web crawler operated by Dispatch Inc., a website monitoring and incident management service founded in 2014. The bot is designed to periodically fetch pages from monitored websites to verify availability, detect downtime, and measure response times for alerting purposes. The product, also named Dispatch, aggregates uptime data and provides real-time notifications to thousands of businesses worldwide, as documented on dispatch.com.

🌐 Technical Behavior

Dispatch crawls target websites at configurable intervals, typically every 60 seconds for critical monitors but can be extended to longer periods (e.g., 5 minutes). The bot uses HTTP/1.1 and HTTPS protocols, sending GET requests to the root URL or specified endpoints. It follows redirects (up to 5 hops) and records HTTP status codes, SSL certificate validity, and page load times. The crawler originates from a set of IP addresses published and maintained at dispatch.com/ips.txt, including ranges 45.33.32.0/20 and 72.14.192.0/18 (as of early 2025). It uses only IPv4 and supports both HTTP and HTTPS connections, but does not fetch external resources like images, CSS, or JavaScript, focusing solely on the requested HTML page. According to official documentation, Dispatch sends requests with a randomized delay between 1 and 5 seconds per host to avoid overwhelming servers, and limits concurrent connections per domain to 2.

📋 robots.txt Compliance

Dispatch strictly honors robots.txt directives. The bot reads the file at the start of each crawl session and will not access any URLs disallowed by the site owner. This is confirmed in the vendor’s public knowledge base article titled “How Dispatch Respects robots.txt” (available at dispatch.com/help/robots). If a Disallow rule prevents crawling, the bot logs a soft error and reports the site as unchecked rather than bypassing the restriction. It also respects Crawl-Delay directives if specified.

🔍 Detection Indicators

The primary User-Agent string is Dispatch/1.0 (full: Mozilla/5.0 (compatible; Dispatch/1.0; +https://dispatch.com/help/bot)). Additional identifiers include the request header X-Dispatch-Bot: true and a custom Via header containing “Dispatch”. The bot also includes a comment line referencing its purpose in the User-Agent field. Webmasters can verify requests by checking if the IP resolves to the dispatch.com domain via reverse DNS (PTR records map to *.dispatch.com). The bot does not use any obfuscation techniques.

📊 Data Usage

Collected data is used exclusively for website monitoring: uptime percentages, response time histograms, and incident alerts. Dispatch Inc. does not use the data for AI training, search indexing, or any secondary purpose. The information is stored temporarily on Dispatch servers and available only to the account owner. Data retention period is 30 days for standard plans, after which monitoring history is aggregated into monthly reports. No full page content is stored or shared with third parties.

⚙️ Rate Limiting Policy

Dispatch is intentionally rate-limited because its default crawl frequency (every 60 seconds) can generate significant load on small servers. The policy rationale is to protect origin servers from accidental overload while still providing near-real-time monitoring. Threshold-based blocking (e.g., limiting to 1 request per 5 seconds per IP) is recommended but not required, as Dispatch already includes randomized delays and respects Crawl-Delay. The bot’s conservative behavior makes it one of the least aggressive monitoring crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.