🤖 Overview

NovaAct is a web crawler operated by Nova Technologies, a company specializing in AI‑driven data analysis and enterprise search. First publicly documented in early 2024 via the official documentation at novaact.ai/crawler, its primary purpose is to collect publicly accessible web content to train and improve Nova’s proprietary large language models and natural language processing systems. The crawled data feeds into the NovaAct AI platform, which offers enterprise‑grade semantic search and summarization services.

🌐 Technical Behavior

NovaAct employs a distributed crawling architecture that originates requests from multiple cloud providers, including AWS, Google Cloud, and Azure. Typical request frequency is 10–20 requests per second per IP, with burst patterns reaching up to 50 requests per second. The crawler uses HTTP/1.1 and HTTP/2 protocols and respects the Accept‑Language header to prioritise English and other major languages. It avoids fetching binary files (e.g., images, videos) unless explicitly required for text extraction, and follows redirects up to five hops. NovaAct publishes its IP ranges in a machine‑readable JSON file at novaact.ai/crawler/ips.json, covering blocks such as 3.0.0.0/9 (AWS), 35.0.0.0/8 (GCP), and 13.0.0.0/8 (Azure). It supports both GET and HEAD requests and maintains a persistent crawl queue with exponential backoff on errors.

📋 robots.txt Compliance

According to the official robots.txt policy published at novaact.ai/robots, NovaAct fully honours Disallow directives and respects Crawl‑Delay parameters. The crawler fetches robots.txt at the start of each crawl session and checks for updates every six hours. No violations have been reported in public security advisories or bug bounty programs, and the company explicitly states that compliance is enforced at the engineering level.

🔍 Detection Indicators

The primary User‑Agent string is NovaAct/1.0 (compatible; Nova Technologies; +https://novaact.ai/crawler). Additional identifying markers include the Nova‑Crawl‑Id HTTP header, which contains a UUID unique to each crawl session, and a fixed order of request headers (e.g., User‑Agent followed by Accept). Behavioral fingerprints include a lack of common browser features (no canvas, no JavaScript execution) and a consistent TLS fingerprint using modern ciphers. The IP ranges listed in the official JSON file can be used to differentiate NovaAct from other bots.

📊 Data Usage

Collected data is used to train Nova’s large language models, improve search relevance in the enterprise search product, and generate training datasets for natural language understanding tasks. The company states it does not collect personal or sensitive information and applies automatic filtering to remove duplicate, low‑quality, or paywalled content. The final datasets are used internally and are not resold.

⚙️ Rate Limiting Policy

NovaAct is rate‑limited because its distributed crawling can consume significant server resources if unrestricted. Recommended thresholds are 100 requests per minute per IP before considering a temporary block, as detailed in the official rate‑limiting guide at novaact.ai/crawler/policy. This policy balances the bot’s legitimate need for data with fair use of server capacity.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.