🤖 Overview

CSHttp is a web crawler and port scanner operated by Censys, Inc., a cybersecurity research company founded by University of Michigan researchers. Its purpose is to systematically index publicly accessible HTTP/HTTPS services across the internet to populate the Censys search engine (censys.io), a platform for discovering internet-connected devices and analyzing TLS certificates, banners, and web technologies. Unlike general search engine bots, CSHttp focuses on security-relevant metadata for attack surface reduction and vulnerability management.

🌐 Technical Behavior

CSHttp performs both TCP port scanning (commonly ports 80, 443, 8080, 8443) and HTTP-level crawling. It follows redirects, collects TLS certificate details, and parses HTTP response headers and server banners. The bot sends requests at a high rate — often multiple per second from distributed IP ranges — using HTTP/1.1 and HTTP/2 protocols. Censys publishes its scanning IP ranges (e.g., 162.142.125.0/24, 167.94.138.0/24) and recommends administrators permit these ranges for accurate asset discovery. The crawler supports IPv4 and IPv6 but does not execute JavaScript or render page content.

📋 robots.txt Compliance

According to Censys’s official documentation at https://support.censys.io/hc/en-us/articles/360059699671-Identifying-Censys-Scans, the web crawler component honors robots.txt Disallow directives by ceasing crawling on any disallowed path. However, the port scanner does not use robots.txt as it operates at the transport layer. The bot first requests /robots.txt to parse rules before crawling each site.

🔍 Detection Indicators

The primary User-Agent string for the web crawler is Mozilla/5.0 (compatible; CensysInspect/1.0; +https://censys.io/robots). The port scanner uses Censys/1.0 and CSHttp/1.0. Behavioral fingerprints include rapid sequential requests to common ports, TLS handshake patterns, and a prefetch of /robots.txt. Source IPs belong to the public Censys network blocks listed in the support article above.

📊 Data Usage

Collected data is stored in the Censys searchable database, used by cybersecurity professionals for asset discovery, vulnerability management, and attack surface reduction. Censys does not use the data for AI training; its primary value is indexing TLS certificates, banners, and web technologies. The dataset is also available via API and researcher tools.

⚙️ Rate Limiting Policy

Because CSHttp generates high-volume requests (frequently scanning the entire IPv4 address space), it is strongly recommended to rate-limit this bot to prevent resource exhaustion and false positives in security monitoring. A threshold-based block (e.g., >100 requests per minute from a single Censys IP) is a practical defense on non-public or sensitive web applications, still allowing legitimate security research scans.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.