🤖 Overview

Nimbostratus is a web crawler operated by the AI research and data platform Nimbostratus Inc., first documented in public access logs around 2022. Its primary purpose is to collect publicly available web content—including text, images, and structured data—which is then used to train and improve large language models and other machine learning systems under the company’s cloud-based AI services. The bot is considered legitimate and is listed on the official Nimbostratus website (nimbostratus.ai/bot) with a dedicated user-agent policy.

🌐 Technical Behavior

Nimbostratus performs multi-threaded HTTP/1.1 and HTTP/2 requests, with a default crawl pattern that mimics a standard web browser by accepting text/html, application/xhtml+xml and following all a and link tags. Its request frequency is moderate, typically 1–5 requests per second per host, with bursts up to 10 requests per second during initial discovery. The crawler originates from a dynamic pool of IP addresses owned by Amazon Web Services (AWS) and Google Cloud Platform, with ASNs such as AS14618 and AS396982. Official documentation (nimbostratus.ai/crawler) confirms it supports If-Modified-Since and ETag headers to reduce redundant crawling. It also sends a custom X-Nimbostratus-Version header with a version string (e.g., 2.1.0) for identification.

📋 robots.txt Compliance

Nimbostratus fully honors robots.txt directives as per the Robots Exclusion Protocol. Its official crawling policy (nimbostratus.ai/robots) states that it will respect Disallow rules with a 60‑second cache for the file. Testing by third‑party researchers (e.g., in 2023 at github.com/nimbostratus/crawler-compliance) confirmed no violations when Disallow: / was set. The bot also obeys Crawl-Delay directives when present, with a default delay of 2 seconds.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Nimbostratus/2.0; +https://nimbostratus.ai/bot), with a secondary string Nimbostratus/1.0 (compatible; robotic crawler) for legacy versions. Behavioral fingerprints include a consistent request pattern of exactly one Accept-Language: en-US,en;q=0.9 header and a Connection: close header on HTTP/1.1 requests. The bot never sends Referer or Cookie headers, distinguishing it from browser traffic.

📊 Data Usage

Collected data is used exclusively for AI training and model fine-tuning within Nimbostratus Inc.’s internal infrastructure, as stated in their privacy policy (nimbostratus.ai/privacy). The content is processed to extract semantic features, relationships, and natural language patterns, which are then incorporated into their proprietary language models. No data is resold or shared with third parties; the company guarantees that raw page content is discarded after 90 days.

⚙️ Rate Limiting Policy

Nimbostratus is rate‑limited because its moderate crawl speed can still impact server performance during peak hours, especially on smaller sites. A threshold‑based block (e.g., >50 requests per minute from a single IP) is recommended to prevent resource exhaustion while still allowing the legitimate crawler to index public content for AI research.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.