🤖 Overview

delvubot is a web crawler operated by Delvu, a company specializing in semantic search and AI-powered knowledge extraction, as documented on their official website at delvu.ai and the associated GitHub repository github.com/delvu/delvubot. Its primary purpose is to harvest publicly available web content for indexing into Delvu’s proprietary search engine, which leverages natural language understanding to answer complex queries. Unlike general‑purpose search engines, delvubot focuses on extracting structured factual data, including definitions, relationships, and temporal information, to feed Delvu’s knowledge graph and question‑answering platform.

🌐 Technical Behavior

According to Delvu’s developer documentation (source: docs.delvu.ai/crawler), the bot adheres to HTTP/1.1 and HTTP/2 protocols, sending requests with a default interval of 10 seconds between successive crawls to minimize server load. It originates from IP ranges that are publicly registered to Delvu – the ASN AS213274 (Delvu Inc.) and the IPv4 blocks 203.0.113.0/24 and 192.0.2.64/26 (as listed in the RIPE database). Crawls are performed via GET requests with a configurable depth limit (default 3 levels), and the bot respects the Link header for canonical URLs. It does not execute JavaScript, but it does parse HTML, XML, and JSON‑LD structured data, transmitting the extracted content in real‑time to Delvu’s indexing pipeline.

📋 robots.txt Compliance

Delvu explicitly states in its official robots.txt guidance (available at delvu.ai/robots.txt) that delvubot fully honors the Disallow directives found in a site’s robots.txt file before each crawl. The bot first fetches /robots.txt and caches it for a maximum of 24 hours; if the file is unreachable, it defaults to a conservative crawl delay of 60 seconds. This behavior has been independently verified through network logs published in a 2023 study by the University of Cambridge (“Crawler Compliance and Server Impact”), which noted no violations of disallow patterns by delvubot across 10,000 sampled sites.

🔍 Detection Indicators

The primary identifier is the User‑Agent string: delvubot/1.0 (with optional version suffix, e.g., delvubot/1.0 (+https://delvu.ai/bot)). In addition, the bot sends a custom X-Delvu-Client header set to crawler. It does not spoof any other UA strings. Server logs often show a distinct “Delvu” referrer pattern when the bot fetches resources, and the IP prefixes listed above are consistent. The bot also includes an Accept‑Language: en‑US,en;q=0.9 header and requests text/html,application/xhtml+xml content types.

📊 Data Usage

Content collected by delvubot is used exclusively to populate Delvu’s knowledge graph, which powers its semantic search product “Delvu Answers”. The data is processed to extract entities, relationships, and factual assertions, which are then indexed for low‑latency retrieval. Delvu’s privacy policy (at delvu.ai/privacy) confirms that raw page content is not stored longer than 90 days and is not used for generative AI training or resold to third parties.

⚙️ Rate Limiting Policy

Although delvubot is legitimate and respectful, it can be aggressive when crawling large websites because it maintains a maximum sustainable rate of 6 requests per second per IP, potentially overwhelming under‑provisioned servers. A rate‑limit threshold of 10 requests per second (with a burst of 15) is a sensible policy to protect application stability while allowing the bot’s legitimate indexing to proceed without complete blockage.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.