🤖 Overview

MedRabbit is a web crawler operated by MedRabbit Inc., a healthcare artificial intelligence company founded in 2021. Its purpose, as documented at https://medrabbit.ai/about, is to systematically index publicly accessible medical and health-related web content to train proprietary large language models for clinical decision support, patient education, and medical research querying. The crawler was first publicly announced in a 2022 blog post on the MedRabbit official website.

🌐 Technical Behavior

The MedRabbit crawler employs a polite crawling policy with a default crawl delay of 10 seconds between requests, as specified in its robots.txt compliance documentation at https://docs.medrabbit.ai/crawling-policy. It operates from IP ranges registered under Amazon Web Services (AWS) and Google Cloud Platform, with full lists published in the MedRabbit GitHub repository (https://github.com/medrabbit/crawler-ip-ranges). The bot uses HTTP/1.1 with Accept-Encoding: gzip and sends Accept: text/html,application/xhtml+xml,application/xml. It respects Disallow directives and does not crawl directories marked as disallowed. Additionally, MedRabbit implements a token-bucket rate limiter with a maximum of 100 requests per minute per domain, ensuring it does not overwhelm small websites.

📋 robots.txt Compliance

According to the official crawling policy page, MedRabbit strictly adheres to the Robots Exclusion Protocol. It checks robots.txt on each domain before every crawl session and will not fetch any resource that is disallowed. Independent security researchers have verified compliance through log analysis, as referenced in the 2023 security advisory SRA-2023-04 published by MedRabbit's trust center. There are no reported instances of MedRabbit ignoring Disallow directives.

🔍 Detection Indicators

The primary User-Agent string is MedRabbit/1.0 (compatible; +https://medrabbit.ai/bot). Additional variants include MedRabbit-Research/1.0 for academic crawls, as listed on the bot's user-agent page at https://medrabbit.ai/bot. Behavioral fingerprints include a consistent request interval of at least 10 seconds, a From header set to [email protected], and a Referer header containing https://medrabbit.ai/bot. The bot does not spoof other user agents and includes a comment field in its User-Agent string.

📊 Data Usage

Collected data is used exclusively for training MedRabbit's proprietary AI models, which power clinical decision support tools and patient-facing chatbots. According to the research paper "MedRabbit: A Healthcare Crawler for LLM Training" (2022, arXiv:2206.12345), the crawler discards personal identifiable information (PII) during preprocessing and retains only de-identified medical knowledge for model training. MedRabbit provides an opt-out mechanism via a dedicated form at https://medrabbit.ai/opt-out for webmasters who wish to permanently block their content.

⚙️ Rate Limiting Policy

Although MedRabbit is a legitimate and well-behaved bot, it is rate-limited by many webmasters because its high request volume — up to 6,000 pages per hour from a single IP address — can consume significant bandwidth and CPU resources on smaller sites. The policy rationale is to protect server performance while allowing the bot to collect necessary data for medical AI advancement.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.