🤖 Overview

CyotekWebCopy is a legitimate, open-source website mirroring tool developed and maintained by Cyotek, a small software company based in the United Kingdom. First released in 2014 and actively updated on GitHub (repository cyotek/WebCopy), its primary purpose is to enable users to download entire websites for offline browsing, archiving, or local development. Unlike search engine or AI crawlers, CyotekWebCopy is an automated, user-driven agent that copies static and dynamic content from a single domain or a set of URLs, often used by web developers, archivists, and researchers to preserve web resources. Its official documentation (available at https://www.cyotek.com/cyotek-webcopy) clearly states that it is designed for personal, non‑commercial use.

🌐 Technical Behavior

CyotekWebCopy operates as a single‑process, multi‑threaded crawler that recursively fetches HTML pages, images, stylesheets, scripts, and other linked assets. By default, it respects the robots.txt file and will not crawl disallowed paths, but this behaviour can be disabled by the user if they explicitly configure the tool to ignore robots.txt (a feature documented in the application’s settings panel). The crawler uses standard HTTP/1.1 GET requests and supports both HTTP and HTTPS. Its request frequency is controlled by the user – there is no fixed rate; the tool allows configuration of concurrent connections (default 4) and a delay between requests (default 0). Without throttling, it can send hundreds of requests per minute, which may appear aggressive to a server. The tool does not use a dedicated IP range; it runs from the user’s local machine, so the originating IP varies per installation. CyotekWebCopy does not send any custom headers beyond standard User‑Agent and Accept fields, and it does not accept cookies by default, though it can be configured to do so. It respects redirections (HTTP 3xx) and can follow them, but will not follow external links outside the target domain unless explicitly told to do so.

📋 robots.txt Compliance

According to the official Cyotek WebCopy documentation and its source code on GitHub (https://github.com/cyotek/WebCopy), the tool respects robots.txt by default. It parses the User‑agent directive for CyotekWebCopy (or * if the specific agent line is missing) and honours all Disallow rules. This compliance is enforced in the RobotsHandler class. However, users can toggle off robots.txt compliance in the application’s advanced settings, which is clearly indicated as a violation of standard crawler etiquette – Cyotek strongly recommends leaving it enabled.

🔍 Detection Indicators

The primary detection indicator is the User‑Agent string, which follows the format CyotekWebCopy/1.0.5.16 (version number varies). The exact string is documented on Cyotek’s website and in the application’s help file. Behaviourally, the tool sends rapid, sequential requests for linked resources (e.g., fetching a page, then its CSS, then its images) without the typical delays of a human user. It does not execute JavaScript, so pages that rely on JS for content may not be fully copied. Additional fingerprints include a lack of Referer header in some cases, and a consistent order of resource fetching (HTML, then CSS, then images). Server logs can also detect a single IP making requests for a high density of unique URLs within a short window.

📊 Data Usage

Data collected by CyotekWebCopy is stored locally on the user’s machine in a folder structure mirroring the original website. The tool is used for offline browsing, website archiving, and local development – for example, to test a site without an internet connection or to analyse its structure. No data is transmitted to Cyotek or any third party; the tool is entirely local and open‑source, with no telemetry or analytics built in. Its GitHub repository explicitly states that the tool “does not phone home”.

⚙️ Rate Limiting Policy

Because CyotekWebCopy can send requests at a high rate if the user configures many concurrent threads and no delay, servers often rate‑limit it to prevent resource exhaustion. This is a legitimate protective measure, not a block; administrators typically throttle connections from a single IP to a few requests per second or require CAPTCHA after a threshold. Cyotek itself recommends that users configure a delay of at least 500 ms between requests and limit concurrent connections to 2–4 to avoid overwhelming servers and to comply with good crawling practices.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.