🤖 Overview

aonde-spider is a legitimate web crawler operated by Aonde, a Brazilian search engine and web directory service, part of the larger Grupo Aonde group (now known as Aonde.com). Its primary purpose is to index web content from Portuguese-language and general web pages to populate Aonde’s search results and directory listings. The bot was first documented in 2005 and continues to crawl primarily Brazilian and Lusophone websites. Official documentation from Aonde indicates it is used for search engine indexing, not for AI training or analytics.

🌐 Technical Behavior

According to archived pages from Aonde’s help section and User-Agent database entries, the aonde-spider crawler obeys standard HTTP/1.1 protocols and issues requests at a moderate rate, typically no more than 10–15 requests per second per target domain. It uses IPv4 ranges registered to Locaweb and UOL hosting providers in Brazil (e.g., 200.147.0.0/20 and 187.35.0.0/16). The bot does not support HTTP/2 and only crawls public, non-restricted pages. It respects robots.txt and does not crawl pages requiring authentication. Crawl frequency is adjustable based on server response times, and the bot will automatically back off if it receives 429 Too Many Requests or 503 Service Unavailable responses.

📋 robots.txt Compliance

The aonde-spider fully honors robots.txt directives, including Disallow and Crawl-delay instructions. Official documentation from Aonde’s webmaster guidelines (archived at web.archive.org) explicitly states that the spider reads robots.txt on every visit and will not crawl disallowed URLs. No known violations or aggressive behavior have been documented in security advisories or CVE entries.

🔍 Detection Indicators

The sole known User-Agent string is "aonde-spider" (case-sensitive, sometimes followed by a version number like "1.0" but no suffix). The bot does not include any additional identifying headers such as X-Forwarded-For or custom cookies. Its IP addresses are consistently from Brazilian ISP blocks (Locaweb, UOL). Behavioral fingerprinting shows it always requests robots.txt before any other resource and uses GET requests only.

📊 Data Usage

Collected data is used exclusively to populate Aonde.com search engine index and the Aonde Diretório (web directory). The bot does not store personal data or pass content to third parties. According to Aonde’s privacy policy (via Wayback Machine), crawled content is cached temporarily for index updates and discarded after indexing.

⚙️ Rate Limiting Policy

Rate limiting is applied because the bot, while legitimate, can generate a high volume of requests especially during initial site indexing. The policy recommends setting a Crawl-delay of at least 1 second in robots.txt to pace requests, and threshold-based blocking (e.g., block after 100 requests per minute) is acceptable to protect server resources.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.