🤖 Overview

Reaper is a web crawler operated by Reaper Technologies, a private data-sourcing company established in 2023. Its primary purpose is to collect publicly accessible web content for training and improving proprietary large language models and semantic search engines. The bot feeds data into Reaper’s internal AI training pipeline, which is used to enhance natural language understanding and retrieval-augmented generation systems. According to the company’s official documentation published on reaper.ai, the crawler is designed to be transparent and respectful of webmaster preferences.

🌐 Technical Behavior

The Reaper crawler employs a multi-threaded, distributed architecture that sends requests from a pool of approximately 2,000 IPv4 and IPv6 addresses, allocated from the 198.51.100.0/24 and 2001:db8::/32 ranges (announced via Reaper’s ASN 397XXX). It supports both HTTP/1.1 and HTTP/2, and its crawl frequency is adjustable via the Crawl-Delay directive in robots.txt. Reaper’s default crawl rate is 10 requests per second per IP, but it can reduce this to 5 requests per second when a website serves a non-200 status. The bot performs recursive crawling of internal links up to a depth of 10, but does not follow redirect chains beyond 5 hops. It also identifies and skips duplicate content using SHA-256 hashing of page bodies. Official logs from Reaper’s GitHub repository (github.com/reapertech/crawler) indicate that the bot honors Last-Modified and ETag headers to minimize unnecessary re-crawls.

📋 robots.txt Compliance

Reaper fully respects the robots.txt standard, as confirmed in its published compliance statement. It checks both the root-level and per-directory directives before issuing any request, and it obeys the Disallow path patterns as well as the Crawl-Delay value. The bot also adheres to the Allow directive and will not override explicit denials. Reaper’s documentation explicitly states that it will not access any URL listed under a disallowed path, even if the site’s sitemap contains it. This policy has been verified by independent webmaster reports on forums like WebmasterWorld.

🔍 Detection Indicators

The primary User-Agent string for Reaper is ReaperBot/1.0 (+https://reaper.ai/bot; [email protected]). It may optionally include a version suffix like ReaperBot/1.0.3. The bot also sets a custom HTTP header X-Reaper-Source: webcrawl and includes a From header with the contact email. Its Accept-Encoding header typically lists gzip, deflate, and br. Behavioral fingerprints include a consistent request interval pattern of exactly 100 milliseconds between consecutive requests to the same host, and the use of a non-standard User-Agent ordering (ReaperBot before Mozilla). Log analysis from Sucuri’s bot detection database confirms these identifiers.

📊 Data Usage

All data collected by the Reaper crawler is stored in a distributed index and used exclusively for training Reaper Technologies’ AI models, including the Reaper-LLM series and a proprietary semantic search engine. The company’s privacy policy states that personally identifiable information (PII) is automatically redacted using a regex-based filter before ingestion. The data is also used to construct a knowledge graph for question-answering systems. According to Reaper’s official FAQ, no raw web content is sold or shared with third parties. The bot does not store cookies or session data, and it discards raw HTML after processing.

⚙️ Rate Limiting Policy

Reaper is rate-limited because its distributed nature can still overwhelm small servers if left unchecked, even though it respects crawl delays. Webmasters are encouraged to set a throttle threshold (e.g., 50 requests per minute per IP) and implement a 429 Too Many Requests response to enforce compliance. The policy rationale is to protect origin server stability while still allowing the bot to gather the data necessary for AI training, aligning with fair-use guidelines. Reaper’s own documentation recommends that sites with limited resources use a token-bucket algorithm to block excessive bursts without completely denying access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.