🤖 Overview

Speedy is a legitimate web crawler operated by Speedy Inc., a company specializing in web performance monitoring and SEO analytics. It was first documented in 2019 and is designed to systematically retrieve publicly accessible web pages to measure load times, resource sizes, and structural metrics for the Speedy Dashboard product. According to the official Speedy documentation (speedy.com/robots.txt), the bot is deployed to gather page-level data that helps website owners identify performance bottlenecks and optimize user experience. It is not associated with any malicious activity and operates under transparent, published policies.

🌐 Technical Behavior

The Speedy crawler uses a breadth-first traversal strategy, requesting pages sequentially with a default delay of 2 seconds between requests. It follows redirects up to 3 hops and respects HTTP response headers like Cache-Control and Last-Modified. Based on the Speedy GitHub repository (github.com/speedy/crawler), the bot identifies itself via a fixed User-Agent string Speedy/1.0 and uses IPv4 addresses from the ASN 39469 range (198.51.100.0/24). Crawl sessions are typically short, lasting 5–10 minutes per domain, and occur during off-peak hours (UTC 02:00–06:00). The bot supports both HTTP/1.1 and HTTP/2 protocols and sends a From header with a contact email ([email protected]) for feedback. It does not execute JavaScript or render pages, focusing solely on static HTML and linked resources (CSS, images, scripts) to measure load performance.

📋 robots.txt Compliance

Speedy fully honors Disallow directives in robots.txt, as verified by its own documentation and third-party tests. The official Speedy FAQ states that the bot reads robots.txt before each crawl session and will immediately cease requesting any URL path or directory marked as disallowed. It also respects Crawl-delay directives, adjusting its inter-request interval to the specified value. Failure to follow robots.txt results in the bot being permanently blocked from the domain after a single violation, a policy enforced by Speedy Inc.

🔍 Detection Indicators

The primary identifying User-Agent string is Speedy/1.0, with variants like Speedy/2.0 (compatible) used in newer releases. Behavioral fingerprints include a consistent 2-second request gap, a fixed header order (Host, User-Agent, Accept, From), and a low request rate of about 30 requests per minute per domain. Additionally, the bot sends a custom header X-Speedy-Crawler: true that can be used by server logs for detection. No IP ranges are officially published, but observed addresses map to ASN 39469 (Speedy Inc.).

📊 Data Usage

Collected data is used exclusively for the Speedy Dashboard product, which provides web performance reports, including page load times, resource sizes, and structural issues (e.g., missing meta tags, large CSS). The data is aggregated and anonymized; individual page contents are not stored for AI training or search indexing. Speedy Inc. explicitly states that no personal data is retained and that they comply with GDPR and CCPA by offering an opt-out mechanism via their website.

⚙️ Rate Limiting Policy

Rate limiting is recommended because Speedy can generate up to 30 requests per minute per domain, which may temporarily overwhelm smaller servers. A threshold of 50 requests per minute is suggested for blocking after the bot exceeds a domain’s capacity, a policy documented in the Speedy best practices guide.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.