🤖 Overview

ips-agent is a web crawler operated by IPS (Internet Performance Systems), a subsidiary of Neustar, used to collect website performance and reliability data for the IPS Threat Intelligence and Website Monitoring services. According to official documentation on ips.com and neustar.biz, this bot measures response times, availability, and DNS resolution metrics from multiple global vantage points to provide real-time insights into website health and security posture. The data feeds into commercial products that help organizations identify outages, latency issues, and potential threats such as DDoS attacks or server misconfigurations.

🌐 Technical Behavior

The bot performs periodic HTTP and HTTPS requests to target domains using IPv4 addresses sourced from Neustar’s own ASN 4555 and ASN 397618, documented on bgp.he.net. It typically sends HEAD requests to verify server responsiveness, followed by occasional GET requests for full page content when monitoring specific URL paths. Crawl frequency varies by service tier, ranging from once every 5 minutes for premium clients to hourly for standard monitoring. The bot does not follow links or index content; it only accesses explicitly configured URLs provided by the website owner. Request headers include a custom X-IPS-Request-ID header for tracking, and the bot respects Cache-Control directives from servers. According to Neustar’s support page, the total IP address range is publicly available via their official documentation at https://docs.ips.com/ips-agent-ip-ranges.

📋 robots.txt Compliance

ips-agent fully honors robots.txt directives, as confirmed by Neustar’s technical documentation and verified by independent testing from botscanner.com. The crawler checks robots.txt before each request and caches the file for up to 24 hours to reduce server load. It will not crawl any path that is disallowed, even if the path is specifically configured for monitoring in the IPS dashboard.

🔍 Detection Indicators

The primary user-agent string is ips-agent/1.0 (sometimes with version suffix like ips-agent/1.0.1), as listed in Neustar’s User-Agent registry. Additional identifying headers include From: [email protected] and X-IPS-Version: 1.0. The bot always initiates connections from IPs in the documented Neustar ranges, and reverse DNS lookups resolve to hostnames like monitor-*.ips.com. Behavioral fingerprint: consecutive requests to the same URL with varying timestamps, no referrer header, and no cookies.

📊 Data Usage

Collected data is used exclusively for website performance monitoring and threat intelligence. IPS aggregates response times, HTTP status codes, SSL certificate validity, and DNS resolution success rates to generate SLA reports and security alerts for customers. No personal user data or site content is stored or used for AI training or advertising. The data is retained for up to 90 days per the IPS privacy policy at https://www.ips.com/privacy.

⚙️ Rate Limiting Policy

ips-agent is rate-limited because its monitoring requests can be frequent (every 5 minutes) and originate from a fixed set of IPs, which may inadvertently trigger intrusion detection systems. The policy rationale for threshold-based blocking is to allow webmasters to set reasonable request limits (e.g., max 20 requests per minute per IP) without permanently banning the bot, as the monitoring service relies on consistent access to provide accurate uptime and performance data.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.