🤖 Overview

Litemage_walker is a web crawler operated by LiteSpeed Technologies as part of the LiteMage caching plugin for Magento (Adobe Commerce). First introduced around 2013, its sole purpose is to pre-fetch and warm the full-page cache of Magento store pages, ensuring that visitors receive lightning-fast responses from the cache. The bot is openly documented in the official LiteMage User Manual and on the LiteSpeed GitHub repository at https://github.com/litespeedtech/litemage.

🌐 Technical Behavior

Litemage_walker crawls using standard HTTP/1.1 requests, respecting the same URL hierarchy as human visitors—product pages, categories, CMS pages, and search result pages. It does not execute JavaScript, submit forms, or follow session-based links. The crawl frequency is configurable via the LiteMage admin panel; default settings can generate hundreds of requests per minute, especially on large catalogues. IP ranges are not published but typically match the server’s own outbound IP or a dedicated crawler IP if assigned. The bot supports compressed responses (gzip/deflate) and sends a User-Agent header that clearly identifies itself. It does not use random delays by default, so aggressive settings can overload shared hosting environments.

📋 robots.txt Compliance

According to the LiteMage documentation (https://docs.litespeedtech.com/litemage/), the crawler honors robots.txt Disallow directives by default. However, administrators are cautioned that blocking the bot from key pages (e.g., product listings) will prevent those pages from being cached, degrading site performance. The crawler can also be controlled via Magento’s admin configuration to ignore robots.txt for specific paths.

🔍 Detection Indicators

The primary User-Agent string is "Litemage_walker" (case-sensitive, no version suffix in most builds). Some deployments may append a version like "Litemage_walker/1.0". The bot does not set custom headers such as X-Forwarded-For or Via. In access logs, it appears with a consistent pattern: frequent requests to non‑static pages, often with a short interval between hits. No SSL certificate validation is performed; the bot requests both HTTP and HTTPS pages.

📊 Data Usage

The data collected by Litemage_walker is used exclusively to populate and refresh the LiteMage full-page cache. No page content is stored permanently, analysed for AI training, or transmitted to third parties. The crawler serves as a performance optimisation tool, reducing server load by ensuring that cache entries are generated during off-peak hours. It does not perform analytics or feed any external data lake.

⚙️ Rate Limiting Policy

Because Litemage_walker can issue hundreds of requests per minute without built‑in throttling, it is frequently rate‑limited by server operators to avoid CPU spikes and database contention. A threshold‑based blocking policy—e.g., allowing 50 requests per minute and then temporarily suspending the IP—is considered best practice to balance cache efficiency with server stability.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.