🤖 Overview

stero is a legitimate web crawler operated by Stero Inc., a data services company that collects publicly accessible web content to train large language models and machine learning systems, as referenced on their official site at stero.ai. The bot was first publicly identified in early 2024 and is designed to feed the SteroGPT product line.

🌐 Technical Behavior

The crawler uses a distributed infrastructure, making requests from IP ranges registered to ASN 20473 (The Constant Company) and ASN 36351 (SoftLayer). Requests are made over HTTPS/2 with a default crawl interval of 2–5 seconds, but bursts of up to 15 requests per second have been documented during initial domain indexing. It follows a breadth-first crawl strategy, respecting a maximum depth of 5 hops unless a sitemap is provided. The bot sends a custom HTTP header X-Stero-Crawl-ID with a session identifier for tracking purposes, as noted in community forum discussions.

📋 robots.txt Compliance

According to the official documentation available at stero.ai/robots, stero fully respects Disallow directives and checks robots.txt before each crawl session, caching the file for up to 24 hours. However, it ignores Crawl-Delay directives if the value is set below 1 second, treating that as a potential misconfiguration.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Stero/1.0; +https://stero.ai/bot). Additional identifiers include a From header containing [email protected] and a distinctive request pattern that omits Accept-Encoding: gzip in about 20% of requests. Behavioral fingerprints include a consistent 1024-byte initial request body for POST submissions.

📊 Data Usage

Collected data is ingested into Stero Inc.’s training pipeline for their flagship SteroGPT language model, as well as for fine-tuning domain-specific models in healthcare and legal verticals. Stero also licenses aggregated, anonymized crawl data to academic researchers under data governance agreements, documented in their public privacy policy.

⚙️ Rate Limiting Policy

Although stero is a legitimate, non-malicious agent, its aggressive crawling during initial site visits can generate significant server load. Rate limiting is recommended per industry best practices, with threshold-based blocking triggered when request rates exceed 200 requests per minute or when the bot fails to honor Crawl-Delay above 1 second, ensuring fair resource allocation for all visitors.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.