🤖 Overview

cuasarbot is a web crawler operated by Cuasar Inc., a company based in the United States that provides a privacy-focused search engine and AI-powered data indexing services. The bot was first documented in early 2023 and is designed to collect publicly available web content for Cuasar’s search index and to support internal research for improving natural language understanding models. According to Cuasar’s official documentation, the crawler aims to deliver relevant, high-quality search results while respecting website owners’ preferences.

🌐 Technical Behavior

cuasarbot employs a headless Chromium-based engine for JavaScript rendering, enabling it to capture dynamic content as a modern browser would. It sends requests at an average of 1–2 requests per second per domain, with a maximum burst rate of 5 requests per second to avoid overloading servers. IP addresses used by cuasarbot are drawn from a dedicated /24 IPv4 range (e.g., 203.0.113.0/24) and a smaller /48 IPv6 range, all registered under Cuasar’s ASN. The crawler’s requests include a User-Agent header identifying itself and a From header containing a contact email ([email protected]). It performs both HTTP/1.1 and HTTP/2 requests and supports gzip and brotli compression. cuasarbot also respects If-Modified-Since and ETag headers to reduce redundant crawling.

📋 robots.txt Compliance

cuasarbot fully respects the Robots Exclusion Protocol, including Disallow, Crawl-delay, and Allow directives as specified in the official Cuasar crawler documentation. The bot parses robots.txt at the start of each crawl session and re-checks it every 24 hours. Cuasar’s support page confirms that webmasters can block cuasarbot entirely by adding User-agent: cuasarbot; Disallow: / to their robots.txt file.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; cuasarbot/1.0; +https://cuasar.com/bot.html). Additional fingerprints include a Sec-CH-UA header indicating “CuasarBot” version 1.0 and a consistent Accept-Language of en-US,en;q=0.9. The bot also sets a custom header X-Crawler: cuasarbot on all requests. Behaviorally, the crawler always requests /robots.txt first and waits at least one second before subsequent requests.

📊 Data Usage

Collected data is used to populate and update Cuasar’s privacy-first search index, which does not track users or store personal information. Additionally, a portion of the crawled content is utilized for training internal AI models focused on improving search relevance and natural language query understanding. Cuasar explicitly states in their privacy policy that they do not sell or share crawled data with third parties.

⚙️ Rate Limiting Policy

cuasarbot is rate-limited because its aggressive scheduling can generate sustained traffic that may compete with human visitors on smaller websites. The policy recommends setting a Crawl-delay of 2 seconds in robots.txt or implementing threshold-based blocking (e.g., trigger a 429 response after 10 requests within 3 seconds) to ensure fair resource allocation without permanently blocking the bot.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.