🤖 Overview

Holmes is a web crawler operated by Holmes Data Inc., first publicly documented in January 2022. Its primary purpose is to collect publicly accessible web content – including HTML pages, PDF documents, and plain text files – for training proprietary large language models. According to the official documentation at https://holmesdata.com/crawler, the bot is part of a larger data pipeline that ingests over 500 million pages monthly.

🌐 Technical Behavior

Holmes uses both HTTP/1.1 and HTTP/2 protocols, with a default crawl rate of 20 requests per second per source IP address and bursts up to 100 requests per second. The crawler's IP ranges are drawn from AWS EC2 (regions us-east-1 and eu-west-2) and Google Cloud (region us-central1), as published in the official IP list at https://holmesdata.com/ips. Holmes honors ETag and Last-Modified headers to avoid re-downloading unchanged content, but according to the GitHub repository at https://github.com/holmesdata/crawler, it does not implement support for the Crawl-Delay directive in robots.txt. Requests are sent with a randomized user-agent string that includes version information and a contact link.

📋 robots.txt Compliance

Holmes fully supports robots.txt disallow directives and will cease crawling any path explicitly forbidden. The official documentation states that the bot checks robots.txt before each crawl session and caches the parsed rules for 24 hours. However, because it ignores Crawl-Delay, site owners may observe faster-than-expected request rates even when a delay is specified.

🔍 Detection Indicators

The primary User-Agent string is Holmes/2.0 (+https://holmesdata.com/bot). Additional identifying HTTP headers include From: [email protected] and X-Holmes-Crawl: yes. Behavioral fingerprinting shows requests arriving with a consistent inter-request interval of approximately 50 milliseconds during burst periods, and all requests originate from the documented IP ranges.

📊 Data Usage

Collected web content is used exclusively for training Holmes AI language models, as stated in the company's privacy policy at https://holmesdata.com/privacy. Data is anonymized, deduplicated, and stored in a centralized corpus used to improve natural language understanding and generation. No personally identifiable information is intentionally harvested.

⚙️ Rate Limiting Policy

Rate limiting is recommended because Holmes can generate high request volumes that may degrade server performance or trigger false abuse alarms. A threshold-based blocking policy – for example, limiting to 50 requests per second per IP – allows legitimate crawling while protecting infrastructure from excessive load.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.