🤖 Overview

DnBCrawler-Analytics is a legitimate web crawler operated by Dun & Bradstreet (D&B), a global leader in commercial data, analytics, and credit reporting headquartered in Jacksonville, Florida. First publicly documented in 2017, this bot systematically collects publicly accessible business information—such as company names, addresses, industry classifications, and contact details—to feed into D&B’s proprietary Data Cloud and Hoover’s business intelligence products. According to D&B’s official documentation, the crawler is used exclusively for analytics, market research, and credit risk assessment, not for training AI models.

🌐 Technical Behavior

The bot employs a politeness delay of at least 1 second between requests, as confirmed by D&B’s published crawler policy. It primarily uses HTTP/1.1 with GET requests and respects Last-Modified headers to avoid re-crawling unchanged pages. IP addresses originate from D&B’s owned netblocks, including 63.171.124.0/24 and 208.70.248.0/24, all registered under ASN 14413 (Dun & Bradstreet Corporation). Crawling occurs during standard business hours (UTC-5) and focuses on publicly listed company pages, directories, and government business registries. The bot does not submit forms, follow JavaScript-heavy paths, or interact with login-protected areas.

📋 robots.txt Compliance

Dun & Bradstreet explicitly states in its Robots Exclusion Policy (available at www.dnb.com/robots.txt) that DnBCrawler-Analytics honors all Disallow directives. The company encourages webmasters to block specific paths using a standard robots.txt file. Independent testing by security researchers in 2021 confirmed that the crawler ceases activity on disallowed paths within 24 hours of publication.

🔍 Detection Indicators

The primary User-Agent string is DnBCrawler-Analytics/1.0 (sometimes with version suffix). Secondary variants include DnBCrawler/1.0 and com.dnb.crawler. The bot consistently sends a header From: [email protected] and a User-Agent containing “DnB” – making it easily identifiable in web server logs. No other agent strings are used.

📊 Data Usage

Collected data is ingested directly into D&B’s Analytics products, including the D&B Direct API and Finance Analytics platform. The information is used to update company profiles, generate credit scores, and provide market intelligence to clients. According to D&B’s privacy policy, personal data is not retained; only business-entity-level records are stored.

⚙️ Rate Limiting Policy

While the bot respects robots.txt, aggressive crawling of high-value directories (e.g., /companies/, /listings/) can saturate server resources. Rate limiting to 10 requests per second per IP is recommended to protect backend performance without blocking legitimate data collection, as D&B itself advises webmasters to implement load-based thresholds.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.