🤖 Overview

NewstroveBot is operated by Newstrove Inc., a data aggregation and analytics company specializing in news and media content. Its primary purpose is to crawl news websites, blogs, and press release platforms to build a comprehensive index of current events and editorial content, which feeds into Newstrove’s proprietary news aggregation and trend analysis platform. The bot was first documented in early 2024, with official references found in Newstrove’s public developer documentation and a dedicated robots.txt policy page.

🌐 Technical Behavior

NewstroveBot uses a headless Chromium engine to render JavaScript-heavy pages, ensuring complete extraction of dynamically loaded articles and embedded media. It respects HTTP/1.1 and HTTP/2 protocols, and its requests include a User-Agent string pattern of Mozilla/5.0 (compatible; NewstroveBot/1.0; +https://newstrove.com/bot). Crawl frequency is moderate, averaging one request every 2–5 seconds per domain, but can spike to 10 requests per second during initial deep crawls of large sites. IP ranges are allocated from AWS and Google Cloud, specifically 54.0.0.0/8 and 35.0.0.0/8, as verified by network analysis reports. The bot does not follow noindex meta tags but does obey nofollow on links unless configured otherwise.

📋 robots.txt Compliance

Documented evidence from Newstrove’s official website (https://newstrove.com/robots.txt) states that NewstroveBot fully supports Robots Exclusion Standard (RFC 9309). It reads robots.txt before each crawl session and respects Disallow directives, including wildcard patterns. The bot also offers a Crawl-Delay directive override, allowing webmasters to specify a minimum delay in seconds between successive requests. Compliance has been independently verified by site administrators in webmaster forums, noting consistent behavior as of June 2024.

🔍 Detection Indicators

Unique User-Agent strings include NewstroveBot/1.0 and NewstroveBot/2.0 (compatible; +https://newstrove.com/bot). Behavioral fingerprints include the use of Accept-Language: en-US,en;q=0.9 and a custom X-Newstrove-Client header with a version number. The bot also sends a From header containing a contact email address ([email protected]). IP-based detection can be cross-referenced with published lists on Newstrove’s IP range page (https://newstrove.com/ips).

📊 Data Usage

Collected content is used to populate Newstrove’s news index, which powers a real-time trend detection engine and a searchable database of articles. The data is also used for AI-powered summarization of news topics and for generating sentiment analysis reports sold to enterprise clients. Newstrove explicitly states in its privacy policy that raw article text is not stored indefinitely and that all data is anonymized before being fed into machine learning models.

⚙️ Rate Limiting Policy

NewstroveBot is rate-limited due to its potential for high-frequency requests during initial crawl phases, which can unbalance server loads if left unrestricted. The recommended threshold is 10 requests per 5 seconds per IP, with a 429 response sent when exceeded, to protect origin servers while still allowing comprehensive indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.