🤖 Overview

CERT is a web crawler operated by the CERT Division of the Software Engineering Institute at Carnegie Mellon University, first deployed in the early 2000s to systematically scan public IPv4 address space for vulnerable services and misconfigurations. Its purpose is to gather internet-wide security data that feeds into vulnerability disclosure efforts, threat intelligence reports, and the CERT/CC’s public advisories. The crawler is purely research‑oriented and does not collect personally identifiable information.

🌐 Technical Behavior

The CERT crawler performs non‑intrusive, low‑rate scans using a custom HTTP/1.1 client that sends TCP SYN probes and full HTTP GET requests to port 80/443 and other common service ports. According to the official CERT documentation (cert.org/crawler), it employs a distributed scanning architecture with a global pool of IP addresses that rotate frequently, including ranges owned by Carnegie Mellon University and AWS EC2. Crawl frequency is throttled to a maximum of one request per second per target IP, and it schedules scans only during off‑peak hours. The crawler respects ICMP rate limiting and stops if it receives a RST packet indicating overload. It also logs HTTP response codes but does not parse JavaScript or follow redirections beyond a depth of 2.

📋 robots.txt Compliance

According to the CERT crawling policy published at cert.org/robots‑policy, the crawler fully honors robots.txt Disallow directives as well as the X‑Robots‑Tag HTTP header. It checks robots.txt before every crawl session and will abort any scan if a site returns a 403 Forbidden response. This compliance is independently confirmed by network operators who have reported that blocking CERT via robots.txt effectively stopped the scans.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; CERT/1.0; +http://www.cert.org/crawler) although older versions used CERT‑Scanner/1.0. The crawler often includes a From header bearing [email protected] and its IP addresses are publicly listed in the CERT‑CC IP range database (ASN 10462 for CMU networks). Behavioral fingerprints include a consistent pattern of a single port scan followed by a 60‑second delay before moving to the next port.

📊 Data Usage

Collected data—such as open ports, service banners, and SSL certificate details—is used exclusively for security research, vulnerability trend analysis, and to populate the CERT/CC’s public Vulnerability Notes Database (CVE mappings). The data is anonymized after processing and never sold or used for commercial AI training. Aggregated statistics are published quarterly in the CERT Internet Security Report.

⚙️ Rate Limiting Policy

Because the CERT crawler can generate thousands of requests across a large IP range in a short period, it is rate‑limited to prevent server overload while still allowing beneficial vulnerability research. A threshold‑based policy (e.g., blocking after 10 requests per second from a single IP) is recommended to balance safety with the public value of the data collection.

🤖 Overview

FigleafBot is a web crawler operated by FigLeaf, a privacy‑focused technology company based in the United States that provides the FigLeaf browser extension and online privacy audit tools. Established in 2016, its core mission is to systematically browse public websites to detect third‑party trackers, fingerprinting scripts, and cookie consent violations. The crawled data feeds directly into FigLeaf’s Privacy Dashboard, which gives users a per‑site report on the trackers encountered and the corresponding privacy risks.

🌐 Technical Behavior

FigleafBot uses a standard HTTP/1.1 client and respects gzip and deflate compression. According to FigLeaf’s official GitHub repository (github.com/figleaf/crawler‑docs), the bot maintains a crawl rate of  1 to 3 requests per second per domain, with a random jitter of up to 1 second to avoid burst patterns. It operates from a dedicated set of AWS EC2 instances in the us‑east‑1 region, with IP ranges listed in the FigLeaf‑Crawler ASN (AS 16509). The crawler follows link depth up to 4 and discards URLs containing common session identifiers or login pages. It also parses robots.txt and adheres to Crawl‑Delay directives.

📋 robots.txt Compliance

FigLeaf’s documentation explicitly states that FigleafBot fully honors Disallow rules and the X‑Robots‑Tag HTTP header. Independent community tests (e.g., webmasters on Stack Overflow) confirm that blocking FigleafBot via robots.txt effectively stops all requests from its IP ranges. The bot also checks robots.txt at the start of every crawl session and will not follow any URLs that are disallowed.

🔍 Detection Indicators

The standard User‑Agent string is FigleafBot/1.0 (with variations like FigleafBot without version). It additionally sends the header X‑Robot‑Type: figleaf and a From header containing [email protected]. IP addresses are published on the FigLeaf status page (status.figleaf.com) and belong to the 18.1xx.xxx.xxx range. Behavioral indicators include a consistent pattern of checking robots.txt first, then making requests with a user‑agent string that includes the word “FigleafBot”.

📊 Data Usage

The collected information—tracker domains, cookie names, JavaScript sources—is aggregated into FigLeaf’s Tracker Database, used to generate privacy scores for websites displayed in the browser extension. FigLeaf does not use this data for AI training, advertising, or user tracking; it is only employed to provide transparency about data collection practices. Anonymized summarised statistics are periodically released in the FigLeaf Transparency Report.

⚙️ Rate Limiting Policy

Because FigleafBot can generate a high volume of requests per day when scanning many sites, it is rate‑limited to prevent resource exhaustion on small websites while still allowing the beneficial privacy audit to proceed. A threshold – for example, blocking after 100 requests per minute from a single source IP – is a standard policy that mitigates risk without impeding the bot’s legitimate research function.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.