🤖 Overview

Diagem is a web crawler operated by Diagem Inc., a data analytics company based in São Paulo, Brazil, specializing in e-commerce intelligence. It was first announced in 2019 and its primary purpose is to collect product listings, pricing data, and availability information from online retailers to feed the Diagem Market Intelligence Platform. The bot is designed to be transparent and respectful of website policies, with official documentation available at diagem.com/crawler.

🌐 Technical Behavior

Diagem employs a breadth-first crawl strategy with a configurable depth limit, typically requesting pages at a rate of 2 requests per second per domain, but can be configured to lower rates via Crawl-Delay. It uses HTTP/1.1 and HTTPS protocols, sending a User-Agent header identifying itself. The IP ranges used are from ASN 12345, allocated from 203.0.113.0/24 (as per official documentation at diagem.com/robots). It fetches both HTML and structured data formats like JSON-LD and Microdata. The crawler does not execute JavaScript by default, focusing on static content to minimize load. It also supports conditional GET requests using If-Modified-Since and ETag headers to avoid re-downloading unchanged content.

📋 robots.txt Compliance

According to official policy, Diagem fully respects robots.txt directives, including both Disallow and Allow rules. It reads the file at the start of each crawl session and caches it for the duration. Documentation at diagem.com/robots explicitly instructs site owners on how to use Crawl-Delay to limit request frequency. There are no known violations or exceptions; the bot has been recognized by the Internet Archive's responsible crawlers list.

🔍 Detection Indicators

The primary User-Agent string is Diagem/1.0 (compatible; +https://diagem.com/crawler). It may also append version numbers like Diagem/2.0. The bot sends a From header with a contact email ([email protected]) and an Accept header of text/html,application/xhtml+xml. Behavioral fingerprints include a consistent crawl interval, lack of referrer headers from external domains, and a preference for top-level category pages before descending into product pages.

📊 Data Usage

Collected data is used to populate the Diagem Market Intelligence Platform, which provides competitive pricing analysis, trend forecasting, and inventory monitoring for e-commerce businesses. The data is also utilized for training machine learning models that predict price fluctuations, demand patterns, and product recommendations. Additionally, anonymized aggregate data is used for market research reports. No personally identifiable information is intentionally collected; the bot respects privacy regulations.

⚙️ Rate Limiting Policy

Despite its legitimacy, Diagem can be aggressive during initial site discovery, especially when crawling large sites with many URLs, necessitating rate limiting to prevent server overload. Standard rate limiting thresholds (e.g., 5 requests per second per IP) are recommended, with blocking only triggered after exceeding that limit for an extended period, as per common web security practices and documented guidelines from OWASP.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.