🤖 Overview

akamai-sitesnapshot is a proprietary web crawler operated by Akamai Technologies, a global content delivery network (CDN) and cloud services provider. Its primary purpose is to periodically capture full-page snapshots of websites for Akamai’s Site Snapshot service, which is part of the Akamai Web Performance & Security suite. The service delivers cached, static visual representations of web pages to enable faster load times for end users, especially on mobile networks or during peak traffic. According to Akamai’s official documentation (docs.akamai.com), Site Snapshot uses this crawler to pre-render pages and store them in Akamai’s edge cache, reducing origin server load and improving time-to-first-byte (TTFB). The bot is not designed for indexing or AI training but for performance optimization.

🌐 Technical Behavior

The akamai-sitesnapshot crawler follows a scheduled crawl pattern configurable by Akamai customers via the Property Manager interface. It typically requests pages using the HTTP/1.1 protocol with a User-Agent string that includes “AkamaiSiteSnapshot” or “Akamai-SiteSnapshot/1.0” and a version number. The crawler can issue requests from Akamai’s extensive IP range, which spans thousands of IPs across different ASNs (notably AS16625, AS20940, AS36183). Akamai’s public IP lists (available via their API at ipgen.akamai.com) indicate the crawler originates from US, EU, and Asia-Pacific edge nodes. The bot respects standard HTTP headers but does not support cookies or JavaScript; it captures only the initial HTTP response (including HTML, CSS, images). It typically sends one request per page per snapshot cycle, with a minimum interval of 30 seconds between requests to the same domain (configurable by the customer). According to Akamai’s support articles, the bot may also send If-Modified-Since and ETag headers to avoid re-downloading unchanged content.

📋 robots.txt Compliance

Akamai’s Site Snapshot documentation explicitly states that the crawler honors robots.txt directives. It will not crawl any path disallowed in the robots.txt file. Additionally, the bot respects the Crawl-Delay directive if present. This is documented in Akamai’s developer guide (developer.akamai.com) under the “Site Snapshot” section, where they advise customers to ensure their robots.txt allows the bot (“Disallow:” or “Allow:” for specific paths). However, the bot may still crawl pages that are disallowed if the customer has overridden default behavior via Akamai’s configuration options, but such overrides are rare and explicitly warned against in best practices.

🔍 Detection Indicators

The primary detection method is the User-Agent string. Known examples include: “Akamai-SiteSnapshot/1.0 (AkamaiSiteSnapshot)” or “AkamaiSiteSnapshot/1.0”. The bot also sends a custom HTTP header X-Akamai-SiteSnapshot set to “true” (though this is not always present). Additionally, the Via header may contain “Akamai-SiteSnapshot” when passing through Akamai’s proxy. The bot’s IP addresses are part of Akamai’s known scanning ranges, which can be identified using Akamai’s IP reputation lists (publicly available at github.com/AkamaiTech/ip-ranges). Behaviorally, the bot typically requests only the main page (no images, CSS, or JS) and does not parse JavaScript. It sends a single request per URL per snapshot cycle.

📊 Data Usage

Collected snapshots are used exclusively for Akamai’s Site Snapshot service, which cache static representations of web pages. The data is not used for AI training, indexing, analytics, or any third-party purpose. According to Akamai’s privacy policy (akamai.com/privacy), snapshot data is stored only on Akamai’s edge servers for a limited time (typically 24 hours) and is purged automatically. The service helps reduce origin server load and improve page load times for real users, especially on slow connections. There is no evidence that the bot stores or indexes data beyond the immediate caching need.

⚙️ Rate Limiting Policy

The akamai-sitesnapshot bot is rate-limited because its per-domain request rate (default 30-second intervals) can still accumulate if multiple customers trigger site snapshots simultaneously. Web application owners frequently apply threshold-based blocking (e.g., >10 requests per second from a single IP) to prevent accidental overload, even though the bot is legitimate. The rationale is that snapshot requests, while low-frequency, can spike during configuration changes or initial setup, impacting origin server performance if not throttled.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.