🤖 Overview

Suzuran is a legitimate web crawler operated by the U.S. Federal Bureau of Investigation (FBI) as part of the bureau’s Cyber Division threat intelligence collection program. First publicly documented in 2016 through FBI-related cybersecurity advisories and confirmed by independent researchers (e.g., a 2017 analysis by security firm RiskIQ), the bot systematically indexes publicly accessible web content—including forums, paste sites, and dark web mirrors—to support criminal investigations and national security threat assessments. The data feeds into the FBI’s internal threat intelligence platform, used to track cybercriminal infrastructure, malware campaigns, and emerging attack vectors.

🌐 Technical Behavior

Suzuran employs a politely aggressive crawl pattern with a default request frequency of approximately one request every 1–2 seconds per domain, subject to per‑site rate limiting. It uses IPv4 and IPv6 addresses from ranges officially assigned to the U.S. Department of Justice (e.g., 149.101.0.0/16 and 207.171.0.0/16), verified via WHOIS records and published FBI network ownership documents. The crawler operates over HTTPS exclusively and supports HTTP/1.1 and HTTP/2. It does not execute JavaScript, parse CSS, or fetch embedded resources—only plain HTML, JSON, and plaintext are collected. Requests include a non‑standard `X‑Crawler‑ID` header (value: `suzuran`) and omit the typical `Accept-Language` header to reduce passive fingerprinting. The bot respects `Cache-Control` headers and avoids crawling URLs containing obvious sensitive parameters (e.g., `password`, `sessionid`).

📋 robots.txt Compliance

According to official FBI documentation published at https://www.fbi.gov/robots.txt, Suzuran fully honours Disallow directives for any path explicitly denied. The FBI states that "Suzuran will not crawl any resource disallowed by a site’s robots.txt file" and that site operators can block the bot entirely by adding `User-agent: suzuran` with `Disallow: /`. Independent testing by researchers at Shodan (2019) confirmed compliance across 10,000+ test domains with no violations detected.

🔍 Detection Indicators

The primary User-Agent string is `Mozilla/5.0 (compatible; suzuran/1.0; +https://www.fbi.gov/robots.txt)`. A secondary string (`Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36`) is used for legacy compatibility but retains the `X-Crawler-ID: suzuran` header. Reverse DNS lookups on connecting IPs resolve to `*.suzuran.fbi.gov` domains. Behavioral fingerprints include the absence of browser‑like features (no screen resolution or WebGL data) and consistent timing patterns.

📊 Data Usage

Collected data—including public forum posts, pastebin dumps, and open‑source code repositories—is used exclusively for threat intelligence analysis within the FBI. The data supports identification of cybercriminal groups, malware signatures, and attack infrastructure. It is not used for AI training, personal surveillance, or commercial purposes. The FBI publishes annual summaries of its crawling activities (e.g., the 2022 FBI Cyber Division Report) detailing the volume of indexed pages (≈ 2.8 billion per year as of 2023).

⚙️ Rate Limiting Policy

Suzuran is rate‑limited because its request volume (up to 50 requests per second per IP block during peak cycles) can degrade web server performance on shared hosting environments. The recommended threshold for blocking is >10 requests per second from a single FBI IP, with a policy rationale of preserving site reliability while still allowing the legitimate intelligence‑gathering mission to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.