🤖 Overview

NewsGatherer is a legitimate web crawler operated by NewsEdge Corporation (now part of Acquire Media) and also associated with Moreover Technologies, a news aggregation and monitoring service. Its primary purpose is to systematically collect publicly available news articles, blog posts, and press releases from thousands of sources worldwide, feeding data into the NewsEdge and Moreover platforms for real-time news alerts, market intelligence, and content aggregation products used by enterprises and media organizations.

🌐 Technical Behavior

The NewsGatherer crawler typically respects standard HTTP protocols and identifies itself via the User-Agent string “NewsGatherer”. According to official documentation from Moreover Technologies, the bot crawls at a moderate rate — making requests every 10 to 30 seconds per site to avoid overloading servers — though this can vary by configuration. IP ranges are not publicly fixed; the crawler uses a set of shared IP addresses belonging to NewsEdge’s data centers, often listed in reverse DNS as “newsgatherer.newsedge.com” or “crawler.moreover.com”. It follows standard HTTP 1.1, supports gzip compression, and requests HTML pages, RSS feeds, and XML sitemaps, but does not execute JavaScript or render dynamic content.

📋 robots.txt Compliance

The NewsGatherer bot has been documented in public robots.txt files and explicitly honors Disallow directives (source: www.robotstxt.org/db/newsgatherer.html). For example, websites can block it entirely with “User-agent: NewsGatherer” followed by “Disallow: /”. There are no known cases of the bot ignoring such rules; its operators (NewsEdge/Acquire Media) state that compliance is enforced to respect publisher policies.

🔍 Detection Indicators

The primary detection signature is the User-Agent string “NewsGatherer” (case-sensitive). Additional behavioral fingerprints include a reduced crawl frequency compared to major search bots, a tendency to request pages during off-peak hours, and a pattern of fetching well-formed URLs with high link depth. Some servers also detect the bot via the X-Forwarded-For header or reverse DNS lookup revealing “newsgatherer*.newsedge.com”.

📊 Data Usage

Collected content is used for real-time news aggregation in services such as NewsEdge’s NewsDesk and Moreover’s Market Intelligence platforms. The data is indexed, categorized, and made searchable for corporate clients, analysts, and media monitoring. While not directly used for AI model training, the aggregated dataset may feed into natural language processing pipelines for sentiment analysis and trend detection (as described in Acquire Media’s product documentation).

⚙️ Rate Limiting Policy

NewsGatherer is rate-limited because its systematic, high-volume crawling can consume significant bandwidth and server resources if left unchecked. A threshold-based approach — blocking IPs that exceed a certain number of requests per minute — is recommended to protect web applications from accidental overload while still allowing legitimate content indexing to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.