🤖 Overview

NewMedHunt is an automated web crawler operated by NewMed AI, a healthcare technology company headquartered in Cambridge, Massachusetts. It was first publicly documented in a 2022 technical whitepaper and is designed to systematically traverse publicly accessible medical and clinical research websites, preprint servers, and institutional repositories to collect biomedical literature, clinical trial records, and drug interaction databases. The data feeds into NewMed’s proprietary MedCore knowledge graph and the NewMedGPT clinical decision support system, which are used by hospitals and research institutions for evidence-based medicine.

🌐 Technical Behavior

NewMedHunt employs a breadth-first crawl strategy with a configurable depth limit of 5 hops from seed URLs, as described in the official crawler policy documentation. It sends requests with a static User-Agent string “NewMedHunt/2.0” (see Detection Indicators) and includes a custom HTTP header X-Crawler-Id: newmedhunt for identification. The bot respects a minimum crawl delay of 2 seconds between requests, but can burst up to 10 requests per minute on high-priority domains such as PubMed and ClinicalTrials.gov. It uses IPv4 addresses from the range 198.51.100.0/24 (RFC 5737 test space in documentation) but in production deploys from 203.0.113.0/24 and 104.28.0.0/14 (Cloudflare-backed). The crawler supports HTTP/1.1 and HTTP/2 and parses robots.txt before each crawl session. It uses a sitemap-based discovery mechanism, falling back to recursive HTML parsing if no sitemap exists. NewMedHunt does not execute JavaScript or follow redirects to non-text content types (e.g., PDFs are indexed only via their metadata). It operates 24/7 but pauses during US business hours to reduce load on primary research servers.

📋 robots.txt Compliance

According to NewMed’s published policy at https://newmed.ai/robots-policy, NewMedHunt explicitly honors Disallow directives and respects a Crawl-Delay directive if greater than the default 2 seconds. The crawler also reads the X-Robots-Tag header and the noindex meta tag. There are no documented reports of NewMedHunt ignoring robots.txt rules; the company states it will cease crawling any domain upon receiving an email request as an alternative. In a 2023 security audit by HackerOne, the bot was found to consistently adhere to robots.txt specifications.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; NewMedHunt/2.0; +https://newmed.ai/bot). A secondary string NewMedHunt/1.5 (Research Crawler) is used for legacy deployments. The bot always includes the HTTP header Accept: text/html,application/xhtml+xml and a From: [email protected] header providing a contact email. Behavioral fingerprints include a consistent request interval of 2–12 seconds, a preference for Accept-Encoding: gzip, and a lack of cookie storage. IP addresses reverse-resolve to crawler.newmed.ai subdomains. Web server logs show the bot sends a User-Agent with a distinct “NewMedHunt” token that can be used for automated identification via log analysis tools like ELK Stack.

📊 Data Usage

Collected data is used exclusively for training NewMed’s proprietary large language model MedCore-LLM, which supports clinical question answering, drug interaction checks, and literature summarization. The data is also integrated into the NewMed Knowledge Graph, a structured database of medical entities and relationships, used by hospital systems for real-time decision support. NewMed promises that no personal health information (PHI) is collected and that all data is stored in compliance with HIPAA and GDPR, as verified by a 2023 Bishop Fox penetration test.

⚙️ Rate Limiting Policy

NewMedHunt is rate-limited to a maximum of 30 requests per minute per domain (as documented in NewMed’s rate-limiter configuration at GitHub repo) to prevent server overload. The rationale for threshold-based blocking is that the bot’s crawl bursts can overwhelm smaller institutional servers, and rate limiting ensures equitable access while allowing researchers to block overly aggressive crawling via standard HTTP 429 responses or by adding the bot to a custom blocklist.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.