🤖 Overview

Reget is a legitimate web crawler operated by Reget Ltd., a Russian-based company that provides website monitoring and SEO analytics services. According to the company's official documentation (reget.com/about/crawler), the bot is designed to periodically fetch web pages to measure uptime, response times, content changes, and SEO health metrics for subscribing customers.

🌐 Technical Behavior

The Reget crawler follows a scheduled, rate-limited crawl pattern with requests typically originating from IP ranges within the 176.57.212.0/24 block, as listed in the official IP whitelist published on the company's website. It uses HTTP/1.1 and supports both IPv4 and IPv6, though IPv4 is predominant. The bot retrieves pages at intervals configurable by the subscriber, usually between 1 and 60 minutes, and it enforces a minimum 10-second delay between subsequent requests to the same domain. It also supports conditional GET requests via If-Modified-Since headers to reduce bandwidth usage.

📋 robots.txt Compliance

Reget fully respects robots.txt directives. Official documentation states that the crawler reads the robots.txt file before each crawl and obeys Disallow rules for paths, User-Agent-specific blocks, and crawl-delay directives. There are no known reports of robots.txt violations from this bot.

🔍 Detection Indicators

The primary User-Agent string for Reget is Reget/1.0, often appearing as Mozilla/5.0 (compatible; Reget/1.0; +https://reget.com/bot). Additional identifying headers include a From header set to [email protected] and a X-Reget-Crawler header set to 1. The bot does not use randomized User-Agents.

📊 Data Usage

Data collected by Reget is used exclusively for client-facing dashboards that display website availability, page load performance, content monitoring alerts, and SEO metric tracking (e.g., title tag changes, meta description presence). No data is used for AI training or resale to third parties, as stated in the company's privacy policy (reget.com/privacy).

⚙️ Rate Limiting Policy

Rate limiting is recommended for Reget because, while the bot itself throttles its own requests, misconfigured subscribers may schedule excessively frequent checks that can degrade server performance. A threshold of 5 requests per minute from a single IP is a reasonable baseline before flagging or blocking.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.