🤖 Overview

Diamond is a web crawler operated by Diamond Technologies Inc., a data aggregation and AI training company based in San Francisco, California. The crawler was first documented in a technical blog post on diamond.io/crawler/announcement (2023) and is used to collect publicly accessible web content for training proprietary large language models (LLMs) and improving the company’s search indexing product, DiamondSearch.

🌐 Technical Behavior

Diamond performs HTTP/1.1 and HTTP/2 requests with a default crawl rate of 5 requests per second per IP, ramping up to 10 requests per second during deep crawls of high-traffic domains. The bot uses a distributed architecture across AWS and Google Cloud data centers, with IP ranges published in the ASN list for AS15169 (Google) and AS16509 (Amazon). It respects Etag and Last-Modified headers to avoid re-crawling unchanged content, and it follows rel="nofollow" link directives. Crawl depth is limited to 10 levels by default, and the bot honors robots.txt crawl-delay directives.

📋 robots.txt Compliance

Diamond explicitly honors Disallow directives as documented in its official robots.txt specification at diamond.io/crawler/robots-policy. The bot pauses for the specified Crawl-Delay value (if provided) before fetching each page. Testing by webmasters has shown that the bot almost never violates a Declared disallow rule, though it may re-check robots.txt every 24 hours.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; DiamondBot/1.0; +https://diamond.io/crawler/bot). A secondary string DiamondSearch/2.0 (compatible; DiamondBot; +https://diamond.io) is used for search indexing. The bot also sends a custom X-Diamond-ID header containing a unique session token. Behavioral fingerprints include a consistent request interval and a preference for HTML pages over binary files.

📊 Data Usage

Collected data is used to train Diamond AI models, including the Diamond-7B and Diamond-70B large language models, and to populate the DiamondSearch index for public querying. According to the company’s privacy policy (diamond.io/privacy), content is processed, filtered, and stored in encrypted clusters in the United States and Europe. No personal identifiable information (PII) is intentionally retained.

⚙️ Rate Limiting Policy

Diamond is rate-limited because it can generate sustained high volumes of requests during deep crawls, which may degrade server performance. A threshold-based blocking at 20 requests per second per IP ensures fair access for other visitors while still allowing the bot to complete its indexing within reasonable timeframes.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.