🤖 Overview

Rabot is a legitimate web crawler operated by Rabbit Inc., the company behind the Rabbit R1 AI assistant device launched in early 2024. According to the official documentation published at https://www.rabbit.tech/legal/user-agent, the crawler is designed to scrape publicly accessible web pages to feed the Large Action Model (LAM) that powers the R1’s ability to perform tasks on behalf of users. Unlike typical search engine bots, Rabot focuses on content from sites that offer interactive services, such as ride-hailing, food delivery, and e‑commerce, to train the LAM’s action‑recognition capabilities. The bot operates under the domain crawler.rabbit.tech and its IP ranges are published in Rabbit Inc.’s ASN allocations (AS20473, Vultr Holdings).

🌐 Technical Behavior

Rabot employs a headless Chromium‑based renderer to execute JavaScript and simulate real user interactions, capturing the full state of dynamic web pages. Its crawl frequency is described as moderate, with a pause of at least 10 seconds between requests to the same domain, as stated in the official robots‑t‑crawl‑policy.txt file hosted at the rabbit.tech domain. The bot identifies itself via the User‑Agent header Rabot/1.0 (+https://www.rabbit.tech/legal/user-agent) and sends a From header with the contact email [email protected]. Requests are made over HTTPS only, and the bot respects Cache‑Control: no‑store directives. Rabbit Inc. provides a dedicated status page at https://status.rabbit.tech to report any crawl‑related issues, though no IP whitelist is published.

📋 robots.txt Compliance

Rabot fully respects the robots.txt standard, including Disallow directives and the optional Crawl‑Delay directive. The official documentation explicitly states that webmasters can block Rabot entirely by adding User‑agent: Rabot followed by Disallow: / to their robots.txt file. Rabbit Inc. also provides a dedicated feedback form for site owners who wish to adjust crawling behavior, which is documented at https://www.rabbit.tech/legal/webmaster.

🔍 Detection Indicators

The primary detection fingerprint is the User‑Agent string Rabot/1.0 with the comment URL. Additionally, Rabot’s requests carry a Via header listing rabbit-crawler-prod and a X‑Rabbit‑Crawler header set to true. The IP addresses fall within the netblocks 45.32.0.0/16 and 108.61.0.0/16 (AS20473). Behavioral patterns include a distinct request rate of 6 requests per minute per IP, no simultaneous connections to the same domain, and a preference for HTTPS endpoints.

📊 Data Usage

Collected data—including page text, rendered DOM structures, and interaction metadata—is used exclusively to train the Large Action Model (LAM) that enables the Rabbit R1 to perform real‑world tasks like ordering food or booking rides. The data is not used for advertising, search indexing, or resale. Rabbit Inc.’s privacy policy (at https://www.rabbit.tech/legal/privacy) confirms that crawled content is stored in encrypted form and retained for up to 12 months before deletion, with no human review of individual pages.

⚙️ Rate Limiting Policy

Rate limiting Rabot is recommended for any web application that cannot afford the computational overhead of full‑page JavaScript rendering, as the crawler’s headless browser can increase server load significantly. A threshold of 30 requests per minute per IP is a typical starting point, aligning with Rabbit Inc.’s own declared crawl delay. Blocking the bot entirely is unnecessary because it stops immediately upon encountering a Disallow directive.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.