🤖 Overview

Asterias is a web crawler operated by Asterias Inc., a data analytics company based in San Francisco, first documented in July 2022 on their official site (asterias.io). Its primary purpose is to collect publicly accessible web content for training proprietary natural language processing models, aiming to improve contextual understanding and nuance in AI systems.

🌐 Technical Behavior

The Asterias crawler uses a distributed architecture with IP ranges published in a GitHub repository (github.com/asterias/bot-ips), primarily from the 198.51.100.0/24 block. It sends an average of 10 requests per second per IP, with bursts up to 50 requests per second during deep indexing sessions. The bot employs HTTP/1.1 and HTTPS, carrying a User-Agent string of Asterias/1.0 (compatible; +https://asterias.io/bot), an Accept-Language: en-US header, and a From: [email protected] email header. Crawl patterns follow a breadth-first strategy with a 0.5-second delay between requests to the same domain, though this delay is not guaranteed during initial discovery phases.

📋 robots.txt Compliance

According to Asterias’ official robots.txt guidelines (asterias.io/robots.txt), the bot fully respects Disallow directives and supports Crawl-Delay fields. Independent verification by the Web Robots Database (webrobots.io) confirms a compliance rate exceeding 98% across tested sites.

🔍 Detection Indicators

The primary identifier is the User-Agent string Asterias/1.0 with a policy link. Additional fingerprints include the From header, a request pattern that always fetches /robots.txt first, and a session cookie _asterias_session.

📊 Data Usage

Collected data trains Asterias’ commercial AI APIs for text summarization, sentiment analysis, and chatbot development. The company claims no intentional collection of personal data and anonymizes any accidentally captured PII.

⚙️ Rate Limiting Policy

Rate limiting is necessary because the bot can produce high request volumes in short bursts, potentially degrading server performance for smaller websites. A recommended threshold of 200 requests per minute per IP preserves service stability while allowing legitimate crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.