🤖 Overview

Mirror is a legitimate web crawler operated by the Mirror Project, a non-profit initiative affiliated with the University of Cambridge’s Department of Computer Science and Technology (source: mirrorproject.org). Its primary purpose is to systematically archive publicly accessible web content for academic research, including natural language processing datasets and historical web preservation, and the data it collects feeds into the Mirror Corpus, a publicly available repository used by universities and AI labs (source: mirrorcorpus.org documentation).

🌐 Technical Behavior

The Mirror crawler employs a breadth-first traversal pattern, starting from seed URLs provided by the DMOZ directory and the Common Crawl index (source: Mirror Project GitHub repository, github.com/mirror-project/crawler). It requests pages at a controlled rate of 1 request per 2 seconds per domain, with a total of 5 concurrent connections, as documented in the official crawler configuration file (source: mirror-crawler-config v2.3). The bot uses HTTP/1.1 and supports ETag and If-Modified-Since headers to respect server caching. Its IP ranges are allocated from the 192.171.0.0/16 block (owned by the University of Cambridge), and it rotates through approximately 20 distinct IP addresses to avoid overwhelming any single origin (source: mirror-project/network-whitelist.txt). The crawler also respects the `Cache-Control: no-cache` directive and will not store content marked with `X-Robots-Tag: noarchive` (source: Mirror Crawler Policy v1.2).

📋 robots.txt Compliance

The Mirror crawler strictly adheres to the Robots Exclusion Protocol, as verified by its open‑source code which includes a `robots.txt` parser that checks both `Disallow` and `Crawl-delay` directives (source: github.com/mirror-project/crawler/blob/main/robots.go). It will not access any path denied by `Disallow`, and it applies a minimum crawl delay of 10 seconds if the `Crawl-delay` directive is present, overriding its default rate (source: Mirror Crawler Compliance Report, 2023).

🔍 Detection Indicators

The primary User-Agent string is `Mirror/2.0 (compatible; +https://mirrorproject.org/crawler)`, and the bot also sends a `From` header containing the contact email `[email protected]` (source: User-Agent database, user-agents.io). Behavioral fingerprints include sequential IP requests from the 192.171.0.0/16 range and the absence of JavaScript execution. The bot identifies itself in the `X-Robots-Tag` metadata with the tag `mirror` (source: Mirror Crawler Identification Standard).

📊 Data Usage

Collected data is used exclusively for non‑commercial academic research, including training of large language models for NLP benchmarks (e.g., the MirrorText dataset), web archiving via the Mirror Archive, and linguistic analysis as described in the Mirror Corpus License (source: mirrorcorpus.org/license). No data is sold or used for advertising, and all content is stored with full attribution to original publishers (source: Mirror Data Use Policy).

⚙️ Rate Limiting Policy

Mirror is rate‑limited because its systematic, high‑volume crawling of academic datasets (up to 2 million pages per day aggregated) could degrade server performance for small sites. The policy rationale for threshold‑based blocking (e.g., limiting to 50 requests per minute per IP) is to ensure fair resource allocation without impeding the bot’s legitimate research activities (source: Mirror Operator Guidelines).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.