🤖 Overview

aggregator:vocus is a legitimate web crawler operated by Vocus Group Limited, an Australian telecommunications and internet service provider. According to Vocus’s official documentation and discussion threads on their community forums, this bot is used as part of their content aggregation and performance monitoring infrastructure, primarily to collect publicly available data for network analytics and customer-facing speed test or latency comparisons. The crawler feeds into Vocus’s internal systems and does not serve third-party AI training models.

🌐 Technical Behavior

Technical analysis from Vocus’s own product pages and third‑party observations (e.g., on security‑oriented mailing lists) indicate the aggregator:vocus bot issues HTTP GET requests at a moderate rate, typically one request every 5–10 seconds per domain. It respects the Connection: close header and uses IPv4 addresses drawn from Vocus’s public ASN (AS4826, AS9444) and some smaller ranges registered to Vocus in Australia. The crawler does not support HTTP/2 and sends requests with a standard Accept header (text/html,application/xhtml+xml). No known fingerprinting or path‑traversal attempts have been reported; it requests only the top‑level pages (e.g., /, /about, /speedtest).

📋 robots.txt Compliance

Vocus’s published policies, referenced in their community wiki, state that the aggregator:vocus bot fully honors Disallow directives in robots.txt. Testing by third parties (e.g., WebmasterWorld discussions from 2022) confirms the bot ceases crawling immediately upon encountering a disallowed path. It does not attempt to bypass restrictions or crawl excluded directories.

🔍 Detection Indicators

The primary User‑Agent string is aggregator:vocus (with no version number). No other identifying headers are added beyond standard ones. Some observers note the crawler sometimes also includes a comment field: aggregator:vocus (https://www.vocus.com.au/aggregator). The bot makes requests from hostnames like crawl-*.vocus.com.au.

📊 Data Usage

Collected data is used exclusively for Vocus’s internal network monitoring, performance benchmarking of their fibre and wireless links, and to populate public speed‑test comparison pages on their website. No data is sold or used for AI training; Vocus’s privacy policy states that crawled content is temporarily cached and not shared externally.

⚙️ Rate Limiting Policy

aggregator:vocus is rate‑limited because its crawling, while legitimate, can consume significant bandwidth if allowed to run unchecked across many pages. A default rate limit of 10 requests per minute per IP is recommended to prevent accidental overload of smaller sites; Vocus itself advises operators to use robots.txt to control access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.