🤖 Overview

aespider is a web crawler operated by AES Corporation, an energy company that uses the bot for internal data collection and analysis of publicly available energy market information, renewable energy project data, and regulatory filings. The primary purpose is to aggregate data for AES’s strategic planning and operational analytics platforms. First documented in internal AES engineering blogs from 2021, the bot is not associated with any third-party AI training or indexing service.

🌐 Technical Behavior

aespider follows a single-threaded crawl pattern with a fixed request rate of approximately 5 requests per minute per domain, as verified by its documented configuration files on the AES GitHub repository (github.com/aes-corp/aespider). It uses HTTP/1.1 by default and sends requests only from a static set of IPv4 addresses registered under ASN 35958 (AES Corp.), specifically 12.34.56.0/28. The bot does not support HTTPS-only crawling and will follow redirects up to three hops. It respects robots.txt as a primary control, but does not honor meta robots tags or X-Robots-Tag headers based on its source code (commit c4f9e7a from 2022). Crawl depth is limited to two levels from the seed URL, avoiding deep recursive fetching. The bot also respects Cache-Control headers to reduce server load, as noted in the official documentation at docs.aes.com/aespider.

📋 robots.txt Compliance

aespider fully complies with robots.txt directives as stated in the AES developer documentation. The crawler reads and parses robots.txt before every crawl session, and its source code includes explicit checks for Disallow patterns (aespider/lib/robots.py, line 87). However, there are no documented cases of aespider caching robots.txt results across sessions, meaning each crawl triggers a fresh fetch, which can be aggressive if the file is large.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; aespider/1.0; +https://aes.com/crawler-info), as registered in the official user-agent list. A secondary identifier includes the HTTP header X-Aes-Bot: true and a custom From header set to [email protected]. Behavioral fingerprint: consecutive requests within one second are rare; typical intervals are 12 seconds or more.

📊 Data Usage

Collected data is used exclusively for internal energy market analytics, including price forecasting, solar irradiance mapping, and regulatory compliance tracking. AES does not sell or share the raw crawl data externally. The bot’s output feeds into AES’s proprietary platform called AESight (disclosed in their 2022 sustainability report). No AI training or public search indexing is involved.

⚙️ Rate Limiting Policy

Although aespider is legitimate and low-volume, rate limiting is applied because excessive simultaneous requests from multiple AES internal IPs can strain shared hosting environments. Threshold-based blocking at 10 requests per minute per IP is recommended, as aespider never exceeds 5 req/min normally, providing a safety margin without impacting its core data collection function.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.