🤖 Overview

Probethenet is a legitimate web crawler operated by the cybersecurity research firm Probe the Net LLC, first documented in 2019. Its primary purpose is to systematically scan publicly accessible web servers and network services to identify open ports, misconfigurations, and unpatched vulnerabilities, feeding data into a proprietary threat intelligence platform used by security teams and ISPs.

🌐 Technical Behavior

The crawler uses both HTTP and HTTPS protocols, typically sending GET and HEAD requests at a rate of approximately 20–50 requests per minute per target. It probes common ports such as 80, 443, 8080, and 8443, as well as non-standard ports for services like SSH, FTP, and databases. IP ranges are drawn from a documented block allocated to Probe the Net, including 192.0.2.0/24 and 203.0.113.0/24 (RFC 5737 documentation addresses) — actual ranges are publicly listed on their domain whitelist page. The crawler employs parallelized scanning with randomized delays to avoid overwhelming hosts, and it respects the Host header for virtual hosting.

📋 robots.txt Compliance

Probethenet fully honors robots.txt directives as confirmed in its official documentation at https://probethenet.com/robots. The crawler parses the Disallow field and will not crawl any paths explicitly forbidden. However, because the crawler’s primary goal is network-level scanning (not content harvesting), it may still probe ports and services that are not governed by robots.txt, which is a standard industry practice for security scanners.

🔍 Detection Indicators

The primary User-Agent string is Probethenet/1.0; secondary strings include Mozilla/5.0 (compatible; Probethenet/2.0; +https://probethenet.com/bot). Behavioral fingerprints include a consistent request pattern of sequential IP probes and a high ratio of HEAD to GET requests. The crawler does not set custom headers like X-Robots-Tag but always includes a valid From header with contact email.

📊 Data Usage

Collected data — including open port lists, service banners, and HTTP response headers — is aggregated into a real-time threat feed used by security operations centers (SOCs) for exposure assessment. According to Probe the Net’s FAQ page, the company also provides anonymized aggregate statistics to research institutions under a non-disclosure agreement, but never stores private content from web pages.

⚙️ Rate Limiting Policy

This bot is rate-limited because its scanning pattern can generate significant logs and may be mistaken for reconnaissance; a threshold of 100 requests per minute from a single IP within the Probe the Net ranges is recommended before applying temporary blocks, allowing legitimate security scans while preventing resource exhaustion.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.