🤖 Overview

Demandbase-bot is a legitimate web crawler operated by Demandbase, Inc., a leader in account-based marketing (ABM) and sales intelligence. Its primary purpose is to collect publicly available web content—including company websites, job postings, news articles, and press releases—to enrich Demandbase’s account-level data platform, which helps B2B organizations identify target accounts, intent signals, and firmographic attributes. The bot was first documented around 2019 and is widely referenced in web analytics tools and security advisories as a known commercial crawler.

🌐 Technical Behavior

Demandbase-bot performs automated crawling across the web at a moderate rate, typically fetching pages sequentially with a request frequency of a few pages per second per IP. It obeys standard HTTP 1.1 protocols and often uses persistent connections. The crawler originates from a set of dedicated IP ranges managed by Demandbase, which are published on their official documentation page at https://www.demandbase.com/about/trust/crawler/. While the exact CIDR blocks are subject to change, known ranges include subnets registered under ASN AS20473 (Vultr) and AS16509 (Amazon AWS), as the bot may leverage cloud infrastructure. The crawler does not execute JavaScript or load external resources, focusing solely on raw HTML content found in the initial document. It respects Cache-Control and Last-Modified headers to minimize redundant requests.

📋 robots.txt Compliance

According to Demandbase’s published documentation, the demandbase-bot fully honors robots.txt directives, including Disallow, Allow, Crawl-delay, and Sitemap instructions. The bot pauses at least the specified delay between requests. Independent tests by webmasters confirm that the bot stops scraping when encountering a disallow rule. Demandbase provides a dedicated feedback form for site owners to request exclusions from crawling.

🔍 Detection Indicators

The official User-Agent string is “demandbase-bot (https://www.demandbase.com/about/trust/crawler/)” — note the lack of a version number. The bot also sends an identifying header X-Demandbase-Bot: true in HTTP requests. Its IPs are publicly listed at the above URL and can be used for allow-listing or monitoring. Request patterns typically show sequential GET requests with no URL parameters beyond the path, and a user-agent that appears in web server logs without variation.

📊 Data Usage

The collected data is used exclusively for Demandbase’s AI-powered account intelligence and intent data products, which include account scoring, lookalike modeling, and target account identification. The content is not used for general search indexing or public display; it fuels internal machine learning models that analyze company attributes, technologies used, and recent initiatives. Demandbase states that no personal data is intentionally collected, only business-related information from non-password-protected pages.

⚙️ Rate Limiting Policy

Demandbase-bot is rate-limited because its moderate crawl volume can still overwhelm smaller sites if left unrestricted. A threshold-based policy (e.g., blocking IPs exceeding 10 requests per second) is recommended to protect server resources while still allowing legitimate data collection for ABM insights. Excessive crawling may also indicate a misconfiguration rather than malice.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.