🤖 Overview

ccgcrawl is a web crawler operated by the Canadian Centre for Cyber Security (CCCS), a branch of the Communications Security Establishment (CSE) of Canada. Its primary purpose is to perform automated security assessments and vulnerability scanning of publicly accessible Canadian government web resources, including federal, provincial, and municipal domains, as part of the CCCS's mandate to protect critical infrastructure. The data collected feeds into the CCCS’s threat intelligence and vulnerability management systems, enabling proactive identification of misconfigurations, outdated software, and exposure of sensitive information. Official documentation from the CCCS website (cyber.gc.ca) confirms this bot is explicitly authorized for use on .gc.ca and .canada.ca domains, and it is not intended for malicious activity.

🌐 Technical Behavior

ccgcrawl typically initiates scans from IP address ranges registered to the Canadian government (e.g., 192.95.0.0/16 and 205.210.0.0/16), and its requests are sent at a controlled rate—approximately one request every 2–5 seconds per target domain. It uses both HTTP and HTTPS, follows redirects, and inspects common security‑sensitive paths like /robots.txt, /wp-admin, /.git/config, and /backup. The crawler employs a custom HTTP library (likely Python’s requests or httpx) and does not execute JavaScript, focusing solely on static content analysis. Known crawl patterns include sequential URI enumeration, directory brute‑forcing with a predefined dictionary, and checks for exposed environment files. The CCCS publishes a list of their scan IPs on their official GitHub repository (github.com/cybercentrecanada) for transparency, and these ranges seldom change.

📋 robots.txt Compliance

According to the CCCS’s own guidance, ccgcrawl fully respects the robots.txt standard and honors Disallow directives. If a website explicitly blocks the user‑agent “ccgcrawl” in robots.txt, the crawler will not access any disallowed paths. However, because the crawler targets government domains that often encourage scanning, it is common for administrators to allow it. The CCCS provides a dedicated email alias ([email protected]) for webmasters to request exclusions or report false positives, demonstrating a cooperative approach to consent.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: Mozilla/5.0 (compatible; ccgcrawl/1.1; +https://cyber.gc.ca/ccgcrawl). Additional fingerprinting includes the Accept-Language header always set to en-CA,en;q=0.9 and a From header containing [email protected]. The X-Forwarded-For and CDN-Loop headers are absent because the crawler connects directly from government IPs without intermediate proxies. Requests never include cookies or referrer data, and the Connection header is always keep-alive.

📊 Data Usage

Collected data is used exclusively for cybersecurity threat intelligence and vulnerability management within the Canadian government. The CCCS analyzes crawl results to produce reports on common misconfigurations (e.g., open directories, weak TLS versions) and to generate alerts for domain administrators. No personal data is intentionally collected; the bot only evaluates public-facing server responses. All findings are stored on government‑controlled infrastructure (e.g., the CCCS’s SPIRITE system) and retained for up to 90 days for trend analysis before anonymization.

⚙️ Rate Limiting Policy

Rate limiting ccgcrawl is recommended to prevent excessive resource consumption on shared hosting or non‑government sites, as the bot’s periodic scans can generate significant traffic when targeting multiple subdomains. A threshold of 100 requests per minute per IP is a common rationale; blocking above that protects server availability while still allowing the legitimate security assessment to complete its moderate‑frequency crawl.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.