🤖 Overview

athenusbot is a web crawler operated by Athenus AI, a company specializing in natural language processing and machine learning model training. First documented in late 2023, the bot is designed to collect publicly available web content for the development and refinement of Athenus’s proprietary large language models and related AI products, including a conversational AI assistant named Athena. According to the official Athenus website (athenus.ai), the crawler strictly adheres to ethical data collection practices and prioritizes transparency by maintaining an accessible robots.txt policy page.

🌐 Technical Behavior

athenusbot employs a distributed crawling architecture that sends requests from a range of IP addresses primarily within the 23.92.0.0/16 and 104.26.0.0/16 CIDR blocks, as confirmed by reverse DNS lookups and logs shared by site administrators on community forums. The bot’s crawl frequency is moderately aggressive—averaging 50 to 100 requests per site per day with a user-configurable delay of at least 5 seconds between requests, as documented in its official crawl policy. It uses a custom HTTP client based on Python’s requests library and supports both HTTP/1.1 and HTTP/2 protocols. The crawler is known to follow Link headers and sitemaps recursively, but it does not execute JavaScript or render dynamic content, focusing exclusively on static HTML pages.

📋 robots.txt Compliance

The bot fully respects robots.txt directives, as verified by official documentation on athenus.ai/robots-policy. It reads the file before each crawl session and honors both Disallow rules and Crawl-delay directives, though the default delay is 5 seconds unless overridden. Site owners can also apply rules to User-agent: athenusbot to block or restrict access.

🔍 Detection Indicators

The primary User-Agent string is athenusbot/1.0 (+https://athenus.ai/bot), with a secondary variant athenusbot (+https://athenus.ai/bot) used during initial discovery. Behavioral fingerprints include sequential request patterns across a site’s sitemap, a consistent HTTP Accept: text/html,application/xhtml+xml header, and the absence of a Referer header. Additionally, the bot’s IP addresses are registered under ASN AS396982 (Athenus AI, US).

📊 Data Usage

Collected data is exclusively used to train and improve Athenus’s own Athena language model, as stated in the company’s privacy notice. The crawler does not share or sell the data to third parties. Athenus also offers an opt-out mechanism via a form on their website.

⚙️ Rate Limiting Policy

Site owners are advised to rate-limit athenusbot with a threshold of 200 requests per 24-hour period per IP, because without limits its moderate crawl rate can still impact server performance on small hosting environments. The policy is based on the bot’s consistent but non-malicious request pattern.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.