🤖 Overview

Industry Cortex Webcrawler is operated by Palo Alto Networks as part of the Cortex Xpanse attack surface management platform (formerly Expanse). Its purpose is to continuously scan public internet-facing assets to discover and inventory an organization’s external attack surface, feeding data into the Cortex Xpanse console for security analysis. The crawler is documented in Palo Alto Networks’ official documentation and knowledge base articles.

🌐 Technical Behavior

The crawler performs TCP port scans and HTTP/HTTPS requests across all public IPv4 address space, focusing on commonly exposed services (e.g., web servers, SSH, RDP). It uses distributed scanning infrastructure with IP ranges assigned to Palo Alto Networks (ASN 396982, as registered in public BGP data). Requests are made at a moderate rate of a few hundred to a few thousand per day per target, but can aggregate from multiple source IPs. The crawler supports both IPv4 and IPv6, and typically probes for TLS certificates, HTTP banners, and application fingerprints. Traffic is sent over standard ports (80, 443, 22, 3389, etc.) and includes full HTTP GET requests with a unique User-Agent.

📋 robots.txt Compliance

According to Palo Alto Networks’ public documentation, the Industry Cortex Webcrawler does not honor robots.txt directives because its scanning is performed as a legitimate security assessment of publicly accessible assets, not as a search indexer. However, organizations can explicitly exclude their IP ranges via the Cortex Xpanse portal or by contacting Palo Alto Networks support. There is no documented evidence of the crawler intentionally ignoring a direct block request from an opt-out list.

🔍 Detection Indicators

The primary User-Agent string is “Cortex Xpanse” (often with a version suffix, e.g., “Cortex Xpanse/1.0”). Additional strings observed include “Mozilla/5.0 (compatible; Xpanse; +https://cortex.paloaltonetworks.com)”. The crawler also sends a custom HTTP header X-Forwarded-For or Via in some cases. Reverse DNS lookups on source IPs resolve to subdomains of xpanse.paloaltonetworks.com. Behavioral fingerprint: the crawler often probes a single IP on multiple ports in rapid succession (within seconds) and then repeats periodically (every few weeks).

📊 Data Usage

Collected data—exposed services, SSL certificates, open ports, and version information—is used to build a real-time inventory of an organization’s external attack surface for the Cortex Xpanse platform. This data is not used for AI training or search indexing but for vulnerability detection, risk scoring, and alerting. Palo Alto Networks states the data is retained only as long as necessary for security analysis and is not shared with third parties.

⚙️ Rate Limiting Policy

Rate limiting is recommended for this crawler because its scanning can saturate low-bandwidth web servers or trigger false positives in intrusion detection systems if thresholds are set too low. The policy rationale is that while the crawler is legitimate, it operates continuously across many targets; applying per-IP rate limits (e.g., 100 requests per minute per source IP) preserves server performance without blocking the essential security assessment function.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.