🤖 Overview

Shelob is an internet-wide scanning crawler operated by Expanse, a Palo Alto Networks company (acquired 2020), designed to continuously map publicly accessible services across all IPv4 space for attack surface management and security risk assessment. Unlike content indexing crawlers, Shelob performs TCP port scanning and HTTP probing to enumerate exposed devices, databases, and web applications, feeding data into Expanse's platform.

🌐 Technical Behavior

Shelob uses distributed scanning with multiple source IPs from ASN 394161 and cloud providers, performing full IPv4 sweeps on common ports (80, 443, 22, 3389) and then making HTTP GET requests with minimal headers and a 10-second timeout. The crawler averages a few hundred requests per second per IP but scales dynamically with documented backoff mechanisms to avoid overwhelming networks, and it also captures TLS certificate details. It supports both HTTP/1.1 and HTTP/2 protocols.

📋 robots.txt Compliance

Per Expanse’s policy at https://expanse.co/robots.txt, Shelob respects robots.txt for HTTP crawling when present, though not for non-HTTP scanning. It fetches robots.txt before each domain crawl and honors Disallow and Allow directives, though occasional non-compliance has been reported due to caching issues.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Shelob/1.1; +https://expanse.co) with version variants including 1.0, 2.0, and 2.1. Additional signals include custom headers X-Expanse-Scan or From: [email protected], and the absence of typical browser headers like Accept-Language. IPs come from ASN 394161 with reverse DNS pointing to *.expanse.co.

📊 Data Usage

Collected data—open ports, service banners, HTTP response bodies, TLS certificates—is ingested into Expanse’s Attack Surface Management platform for asset inventory and risk mapping. It is used for security vulnerability prioritization, breach detection, and compliance monitoring, not for AI training or search indexing. Subscribers access results via dashboard or API.

⚙️ Rate Limiting Policy

Rate limiting is applied because Shelob’s aggressive scanning can saturate servers and trigger false IDS alarms; thresholds typically block requests exceeding 100 per second from a single IP, and administrators can whitelist Expanse IP ranges for legitimate security research.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.