🤖 Overview

LoadTimeBot is a legitimate web crawler operated by LoadTime, a website performance monitoring and optimization service headquartered in the United States. Its primary purpose is to systematically crawl client websites to measure page load times, resource sizes, and overall performance metrics for use in LoadTime’s monitoring dashboards and real-time alerting systems. The bot was first deployed in 2011 and is documented on LoadTime’s official bot information page at https://www.loadtime.com/bot.

🌐 Technical Behavior

LoadTimeBot implements a configurable crawl interval, typically ranging from 5 to 60 minutes depending on the client’s monitoring plan. Each crawl request simulates a full browser session by fetching the entire page including CSS, JavaScript, images, and external resources to accurately measure loading performance. The bot uses HTTP/HTTPS protocols and supports both desktop and mobile user-agent strings when configured. IP addresses originate from LoadTime’s own server infrastructure, which is not publicly documented but can be resolved via reverse DNS lookups on their known monitoring nodes. The bot follows redirects and logs status codes, content sizes, and timing data for every request.

📋 robots.txt Compliance

According to LoadTime’s official documentation, LoadTimeBot fully respects the robots.txt file and will not crawl any disallowed paths. The bot also honors the Crawl-Delay directive if present, slowing its request frequency to the specified value. Compliance is enforced via a built-in parser that re-reads the file on each crawl cycle.

🔍 Detection Indicators

The primary User-Agent string is LoadTimeBot/1.0 with the pattern Mozilla/5.0 (compatible; LoadTimeBot/1.0; +http://www.loadtime.com/bot.html). Some older versions may omit the Mozilla prefix. Identifying headers include a custom X-LoadTime-Crawl-ID header containing a unique session identifier. No other known User-Agent variants exist.

📊 Data Usage

Collected data is exclusively used for LoadTime’s performance monitoring services, including real-time page speed metrics, waterfall charts, historical trend analysis, and threshold-based alerts. LoadTime does not share crawl data with third parties for AI training or advertising. Customers can view aggregated and per-page performance reports via the LoadTime dashboard.

⚙️ Rate Limiting Policy

LoadTimeBot is designed to be respectful of server resources, but its configurable crawl frequency can become aggressive if set to the shortest interval. Rate limiting is recommended to prevent unnecessary load, as the bot generates real browser-like requests that consume bandwidth and CPU time, especially on pages with heavy assets.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.