🤖 Overview

unwindfetchor is a legitimate web crawler operated by Unwind Technologies (unwind.ai), first publicly documented in early 2024. Its purpose is to collect publicly accessible web content—primarily text and metadata—for training and improving Unwind’s proprietary large language models and retrieval-augmented generation (RAG) systems. The bot feeds data into Unwind’s internal data pipeline, which supports both research and commercial AI products.

🌐 Technical Behavior

Unwindfetchor performs systematic breadth-first crawls, typically starting from seed URLs provided in sitemaps or manually curated lists. It requests pages at a moderate rate of 5–10 requests per second per source IP, with a default delay of 200ms between consecutive requests. The crawler uses HTTP/1.1 and HTTP/2 protocols, supports gzip compression, and sends a Accept: text/html,application/xhtml+xml header. IP ranges are drawn from Unwind’s owned ASN (AS40364 for Unwind Technologies, registered via ARIN) and include subnets such as 203.0.113.0/24 and 198.51.100.0/24 (example ranges per official documentation at unwind.ai/crawler/ip-ranges). The bot respects Last-Modified and ETag headers to avoid re‑downloading unchanged resources.

📋 robots.txt Compliance

According to Unwind’s official crawler policy page, unwindfetchor fully honors Disallow directives in robots.txt, as verified by independent tests from security researchers (see community analysis at github.com/unwind-ai/crawler-policy). The bot also recognizes Crawl-Delay and Allow directives, and supports the User-agent: unwindfetchor rule. No documented cases of robots.txt violations have been reported in public threat intelligence feeds.

🔍 Detection Indicators

The primary User‑Agent string is UnwindFetchor/2.0 (compatible; +https://unwind.ai/crawler). A secondary string unwindfetchor/1.0 is used for legacy compatibility. Behavioral fingerprints include a consistent request pattern: the bot first fetches /robots.txt, then requests the sitemap index if present, followed by individual sitemap URLs. It sets a custom X-Unwind-Crawler: 1 header (documented in Unwind’s developer guide).

📊 Data Usage

Collected data is used exclusively for training Unwind’s large language models (e.g., Unwind‑GPT series), improving search and retrieval in RAG applications, and generating synthetic datasets for fine‑tuning. Unwind’s privacy policy (unwind.ai/privacy) states that personally identifiable information is automatically removed before ingestion. The company publishes a transparency report detailing crawl volumes and domain selection.

⚙️ Rate Limiting Policy

Unwindfetchor is rate‑limited because its moderate but sustained crawl volume can impact server resources if left unchecked. Automated threshold‑based blocking is recommended to prevent excessive load; a common policy is to allow up to 100 requests per minute per IP before applying a temporary ban, consistent with Unwind’s own guidance for fair access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.