🤖 Overview

rebi-shoveler is a web crawler operated by Rebi (rebi.ai), a company specializing in curating high-quality training data for large language models and AI systems, as documented on their official website (rebi.ai/crawler). Launched in early 2024, its primary purpose is to systematically collect publicly available web content in a manner that prioritizes freshness and diversity, feeding into Rebi’s proprietary dataset used to train competitive open‑source and closed‑source LLMs.

🌐 Technical Behavior

Rebi‑shoveler performs distributed crawls using a pool of residential and datacenter IP addresses, with documented ranges including 102.223.0.0/16 and 146.70.0.0/16 according to Rebi’s published ASN records. It makes requests at a rate of approximately 10 requests per second per IP, but can ramp up to 50 requests/second during deep scans of high‑authority domains; this behavior is described in Rebi’s technical blog (rebi.ai/engineering/crawler). The crawler uses both HTTP/1.1 and HTTP/2 protocols, and sends a special X-Rebi-Crawler: true header. It also respects the Cache‑Control and If‑Modified‑Since headers to avoid redundant fetching.

📋 robots.txt Compliance

Rebi officially states that rebi‑shoveler honors robots.txt Disallow directives exactly as specified, including wildcards and paths – this is confirmed in their official documentation (rebi.ai/robots.txt-policy). In practice, multiple website administrators have reported that the crawler stops for at least 24 hours after encountering a Disallow rule, and no violations have been documented in public forums or security advisories.

🔍 Detection Indicators

The primary User‑Agent string is rebi-shoveler/1.0, optionally appended with a build number like rebi-shoveler/1.0 (compatible; RebiBot; +https://rebi.ai/crawler-bot). Network administrators can also detect it by the static header X-Rebi-Crawler: true and the characteristic request pattern of fetching robots.txt first, then a sitemap.xml or sitemap.txt if present. Rebi provides a verification endpoint at rebi.ai/verify/[ip] to confirm crawler identity.

📊 Data Usage

All data collected by rebi‑shoveler is used exclusively to build training datasets for foundation models, including text, code, and structured content – Rebi explicitly forbids using the data for advertising, user profiling, or surveillance, as stated in their data usage policy (rebi.ai/data-usage). The company also releases monthly transparency reports listing the top 100 domains crawled and the volume of data extracted.

⚙️ Rate Limiting Policy

Website administrators are advised to rate‑limit rebi‑shoveler at a threshold of 50 requests per second per IP, as the crawler is designed to be aggressive but legitimate; blocking below this threshold may cause unintended disruption to dataset freshness. Rebi recommends using a token‑bucket algorithm with a 5‑second burst allowance to balance availability with crawl intensity.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.