🤖 Overview

The CyberPatrol bot is a web crawler operated by Forcepoint (formerly CyberPatrol LLC and SurfControl) as part of its content filtering and web classification system. Its primary purpose is to systematically scan websites and assign categories (e.g., adult, gambling, social media) to build and maintain the comprehensive URL database used by CyberPatrol filtering software deployed in schools, libraries, and enterprises.

🌐 Technical Behavior

The crawler issues standard HTTP/1.1 GET requests at a moderate rate, typically no more than 2–3 requests per second per domain, and does not follow redirects aggressively. It uses persistent connections but does not perform JavaScript rendering or execute dynamic content, focusing only on static HTML pages and robots.txt files. The IP ranges originate from Forcepoint’s corporate blocks (e.g., 151.145.0.0/16) and are not publicly enumerated in detail; however, reverse DNS lookups reveal hostnames under the forcepoint.com domain. The bot adheres to a crawl delay of at least 1 second between requests when specified in robots.txt, and it respects HTTP 429 (Too Many Requests) responses.

📋 robots.txt Compliance

Official documentation from Forcepoint confirms that the CyberPatrol bot fully honors Disallow directives in robots.txt, including wildcard patterns and path-based exclusions. The crawler fetches robots.txt at the start of each crawl session and caches it for the duration of the visit, never accessing disallowed paths even if the sitemap suggests otherwise.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; CyberPatrol/1.0; +http://www.cyberpatrol.com); an alternative older string is CyberPatrol SiteCat WebBot. Other identifying headers include a From header set to [email protected] and a consistent Accept: text/html,application/xhtml+xml value. Behavioral fingerprints include a lack of Accept-Encoding for gzip and a fixed Connection: keep-alive preference.

📊 Data Usage

All collected URLs are classified into one of over 50 categories using automated heuristics and human validation, then stored in Forcepoint’s centralized threat-intelligence database. This data feeds the CyberPatrol filtering engine, allowing real-time blocking or allowance of web content based on organizational policies. No personal user data is extracted, and the crawl data is never used for advertising or profiling.

⚙️ Rate Limiting Policy

Rate limiting is necessary because the CyberPatrol bot operates continuously across millions of domains, and without throttling it could overwhelm under-provisioned web servers. Threshold-based blocking (e.g., >5 requests per second or repeated 503 responses) protects site stability while still allowing legitimate categorization updates.

sitecat;

🤖 Overview

The SiteCat bot is a legitimate web crawler operated by SiteCat, a company specializing in website monitoring, uptime tracking, and content change detection. It systematically scans websites to detect modifications, broken links, and performance anomalies, feeding data into the SiteCat dashboard used by webmasters and SEO professionals.

🌐 Technical Behavior

SiteCat employs a distributed crawl architecture using multiple virtual machines across AWS regions, sending requests from IP ranges documented under the ASN 16509 (Amazon Web Services). It issues HTTP/1.1 and HTTP/2 requests with a default crawl interval of 60 seconds per monitored URL, but it can be configured by users to check as frequently as every 5 minutes. The bot does not execute JavaScript or parse CSS, but it does download and analyse page source, headers, and metadata. It respects Cache-Control headers and uses conditional GET with If-Modified-Since to reduce bandwidth consumption.

📋 robots.txt Compliance

SiteCat's official documentation explicitly states that the crawler abides by all Disallow and Crawl-Delay directives in robots.txt. The bot fetches robots.txt before each crawl and rechecks it every 24 hours based on the file’s TTL. No violations of robots.txt have been reported in public security advisories or webmaster forums.

🔍 Detection Indicators

The standard User-Agent string is Mozilla/5.0 (compatible; SiteCat WebBot/2.0; +http://www.sitecat.com). Other strings include SiteCat/1.0 for legacy versions. The bot sets a custom X-SiteCat-Bot: true header optionally and includes a User-Agent field with the exact product version. It never masks its identity and does not spoof other browser user agents.

📊 Data Usage

Collected data — including page change timestamps, HTTP status codes, response times, and HTML structure hashes — is stored in SiteCat’s cloud platform and presented to subscribing users via a web interface and API. The data is used exclusively for monitoring and alerting; it is not sold to third parties, used for AI training, or aggregated into public datasets. SiteCat’s privacy policy confirms no personal information is extracted from crawl results.

⚙️ Rate Limiting Policy

Rate limiting is applied because SiteCat may send up to one request per second per monitored site when multiple URLs are being tracked, which could impact small servers. Threshold-based blocking (e.g., >10 requests per minute or high connection concurrency) is recommended to prevent performance degradation while allowing legitimate monitoring activity.

webbot;

🤖 Overview

The WebBot crawler is a legitimate automated agent operated by WebBot LLC, a company that provides web scraping, data extraction, and SEO analysis services. Its purpose is to collect publicly available web content for use in client-facing dashboards, competitor analysis, and market research reports, all delivered through the WebBot API and online platform.

🌐 Technical Behavior

WebBot uses a rotating pool of datacenter IP addresses primarily sourced from OVH and DigitalOcean (ASNs 16276 and 14061), with occasional use of residential proxies for geo-targeted scraping. It sends HTTP/1.1 requests with a configurable crawl rate; the default is 1 request every 2 seconds per domain, but paying customers can increase that up to 10 requests per second under a fair-use policy. The bot supports both HTTP and HTTPS and can handle pagination by following tags. It does not render JavaScript unless explicitly enabled by the user via the API.

📋 robots.txt Compliance

WebBot’s official terms of service state that the crawler respects Disallow directives in robots.txt, though it optionally allows users to override this for proprietary data extraction under a paid license. By default, it fetches robots.txt before each batch crawl and honors Crawl-Delay values. Public discussions on WebmasterWorld indicate no systematic violations of robots.txt by the base version.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; WebBot/1.0; +http://www.webbot.com). An alternative string WebBot/2.0 (compatible; WebBot; +http://www.webbot.com) is used for newer releases. The bot does not send any custom X-headers by default, but it always includes a From header with [email protected]. Behavioral fingerprints include a preference for text/html over other content types and a fixed request ordering (timestamps increase monotonically).

📊 Data Usage

Collected data — including full page HTML, metadata, and link structures — is stored in WebBot’s encrypted cloud storage and made available to clients via REST API endpoints and CSV exports. The data is used for competitive intelligence, price monitoring, and content aggregation. WebBot explicitly states that no user data, cookies, or personally identifiable information is collected, and the scraped content is not used to train large language models or sold to data brokers.

⚙️ Rate Limiting Policy

Rate limiting is necessary because WebBot’s high-frequency crawls, especially under premium plans, can generate significant load on target servers. Threshold-based blocking (e.g., >15 requests per minute or high bandwidth consumption) ensures fair usage across web properties while allowing the bot to complete its legitimate indexing tasks without abusing server resources.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.