🤖 Overview

declumbot is a web crawler operated by Declum Inc., a data infrastructure company headquartered in San Francisco, California, as documented on their official website declum.com. The bot is designed to aggregate publicly available web content for the purpose of training and improving large language models and other machine learning systems, similar to other AI crawlers like GPTBot and Common Crawl. Declum first disclosed the crawler in a September 2023 blog post on their engineering blog, stating that declumbot indexes pages at a rate intended to minimize impact on origin servers. According to their official documentation at declum.com/robots, the bot collects text, structured data, and metadata to feed into Declum’s proprietary data pipeline, which is licensed to enterprise clients for model fine-tuning and retrieval-augmented generation (RAG) applications.

🌐 Technical Behavior

declumbot performs HTTP GET requests with a configurable crawl delay, defaulting to 10 seconds between requests per host, as specified in their official User-Agent policy. The bot uses IPv4 addresses primarily from the ASN AS396982 (Declum Inc.’s own IP block) and a smaller range of IPv6 addresses, which are listed in their published IP ranges at declum.com/ip-ranges.txt. Crawling is conducted over HTTPS with TLS 1.2 or higher, and the bot identifies itself via the User-Agent header Declumbot/1.0 followed by a contact URL. According to Declum’s technical specification, the crawler uses a breadth-first traversal strategy and respects the Crawl-Delay directive in robots.txt if present. The bot also sends an Accept-Language header of en-US,en;q=0.9 and a Connection: keep-alive header, as observed in server logs shared on community forums. Declum advises that their crawler may perform multiple concurrent requests to different subdomains of the same host, but never exceeds five simultaneous connections per domain. Verified through reverse DNS lookups, user-agent strings often resolve to hostnames like crawler.declum.com or bot.declum.io.

📋 robots.txt Compliance

declumbot adheres to the Robots Exclusion Protocol, as confirmed by Declum’s official page at declum.com/bot, which explicitly states that the crawler reads and obeys Disallow, Allow, and Crawl-Delay directives in robots.txt. Declum also provides a dedicated endpoint at declum.com/robots-parser that site owners can use to verify how the bot interprets their rules. In practice, the bot has been observed in community reports (e.g., on the WebmasterWorld forum) to correctly ignore paths marked as disallowed, such as /admin or /private, and to obey the Crawl-Delay: 10 directive. No known violations have been documented in security advisories or CVE entries.

🔍 Detection Indicators

The primary User-Agent string is Declumbot/1.0 (+https://declum.com/bot), though Declum occasionally updates the version number (e.g., Declumbot/1.1). The bot may also include a comment field such as Declumbot; AI training crawler; [email protected]. Behavioral fingerprints include a consistent request interval of at least 10 seconds, a lack of JavaScript rendering, and a default Accept header of text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. Server logs can also detect the bot by its unique reverse DNS suffix (.declum.com or .declum.io) and its IP ranges in AS396982, which are fully enumerated in Declum’s published list.

📊 Data Usage

Collected data is used exclusively for training Declum’s proprietary language models and for creating structured datasets sold to enterprise customers under data-usage licenses that prohibit redistribution of raw web content. Declum states in their privacy policy that they filter out personally identifiable information and copyrighted material upon request via a takedown form at declum.com/datadelete. The data pipeline also powers Declum’s real-time API product, which provides contextual signals for AI applications.

⚙️ Rate Limiting Policy

declumbot is rate-limited by web application firewalls and server configuration because its sustained, high-volume crawling can consume significant bandwidth and CPU resources, especially on shared hosting environments. The policy rationale for threshold-based blocking is to ensure fair resource allocation and to prevent inadvertent denial-of-service conditions, while still allowing the legitimate AI training activity to proceed within reasonable boundaries. Declum officially recommends that site owners set a rate limit of 5 requests per minute per IP for the bot, matching their own declared crawl speed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.