🤖 Overview

cizilla is a legitimate web crawler operated by the SEO analytics platform Cizilla, a service that provides website auditing, backlink monitoring, and search engine ranking analysis. First publicly documented in 2018, the bot systematically scans publicly accessible web content to feed Cizilla’s proprietary index, which powers tools like site health reports, competitor analysis, and keyword tracking. According to the official Cizilla documentation at docs.cizilla.com, the crawler is designed exclusively for collecting non‑private, publicly available URLs and metadata, and it is explicitly not used for AI training, ad targeting, or any form of user profiling.

🌐 Technical Behavior

The cizilla crawler employs a breadth‑first traversal pattern, typically starting from a submitted seed URL and following href and src attributes in HTML documents. It sends an average of 1–3 requests per second per domain, with burst rates capped at 5 requests per second to minimize server load. IP addresses originate from a dedicated block (e.g., 45.76.0.0/16 and 104.238.0.0/16, as listed in its published IP range at github.com/cizilla/ip-ranges). The bot uses both HTTP 1.1 and HTTPS, and it respects the Cache‑Control header to avoid re‑fetching unchanged resources. It also parses sitemap.xml files for structured crawling and collects meta‑robots tags beyond the Robots Exclusion Protocol.

📋 robots.txt Compliance

Cizilla fully supports the Robots Exclusion Standard (RFC 9309). Official documentation at docs.cizilla.com/robots states that the bot reads Disallow directives on every crawl and does not access paths blocked by User‑agent: cizilla. However, it does not honor Crawl‑Delay instructions, relying instead on its own rate‑limiting algorithm to adapt to server response times. Evidence from multiple website operator reports confirms no violations of explicit disallow rules.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; cizilla/1.0; +https://cizilla.com/bot). A secondary agent, cizilla‑mobile/1.0, is used for mobile‑rendered pages. The bot also sends a custom HTTP header X‑Crawler‑ID: cizilla and a Referer header of https://cizilla.com. Its request fingerprint includes a consistent TLS fingerprint of cizilla‑1.0 (JA3 hash: 51c3c1a3...), which can be used for identification in web server logs.

📊 Data Usage

Collected data—page titles, meta descriptions, headings, internal and external link structures, and HTTP status codes—is aggregated into Cizilla’s analytic reports. The platform does not store full page content or user‑specific information; it only retains metadata for 90 days, after which it is anonymized or deleted. The primary purpose is to provide website owners with insights into their SEO performance, backlink profiles, and crawl‑error diagnostics.

⚙️ Rate Limiting Policy

Rate limiting is applied because cizilla, while respectful of robots.txt, can still generate significant request volumes (up to 1,800 requests per hour per domain). Administrators are advised to set thresholds at 10 requests per minute from its known IP blocks to prevent inadvertent server strain, and the bot’s documentation explicitly recommends this approach to balance thorough indexing with server stability.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.