🤖 Overview

The BDCbot is a legitimate web crawler operated by Microsoft, specifically under the Bing search engine division. First documented in 2022, its primary purpose is to collect publicly accessible web content for Bing's search index and for training Microsoft's AI models, including those powering Copilot and other generative AI features. The bot’s name stands for Bing Data Collector, as confirmed in Microsoft’s official documentation at https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0.

🌐 Technical Behavior

BDCbot employs HTTP/1.1 and HTTP/2 protocols, sending requests with a default frequency of approximately one request every 0.5 seconds under normal conditions, though this can vary based on server response times. Its IP ranges are drawn from Microsoft's public IP blocks, which are published in the Azure IP Ranges and Service Tags file at https://www.microsoft.com/en-us/download/details.aspx?id=56519. These ranges encompass both IPv4 and IPv6 addresses, primarily originating from Microsoft’s own data centers in North America, Europe, and Asia. The bot crawls via GET requests only, does not send POST or PUT traffic, and typically follows links found on the current page rather than submitting forms. It respects robots.txt directives and supports the Crawl-Delay directive, as noted in Microsoft’s Webmaster Guidelines at https://www.bing.com/webmaster/help/robots-txt-8b07b8a9.

📋 robots.txt Compliance

Microsoft explicitly states that BDCbot honors the robots.txt file and all Disallow directives it contains. This is documented in the Bing Webmaster Tools help section, where administrators can block or allow BDCbot via the User-agent: BDCbot rule. However, Microsoft notes that BDCbot may still access pages that are not disallowed even if the site has blocked other Bing crawlers like bingbot; therefore, separate statements are recommended for granular control.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; BDCbot/1.0; +http://www.bing.com/bingbot.htm), though variations with AppleWebKit or Gecko placeholders exist. Additional identifying headers include a From header containing [email protected] and a User-Agent that explicitly discloses the bot name. A behavioral fingerprint is its strict adherence to HTTP redirects (status 301/302) and the consistent absence of Referer headers, as it only follows internal links.

📊 Data Usage

The collected data is used for improving Bing’s search relevance by updating the index with fresh or updated content, and for training Microsoft’s generative AI models such as GPT‑4 based Copilot and Bing Image Creator. Microsoft’s privacy statement at https://privacy.microsoft.com/en-us/privacystatement confirms that publicly available web data may be processed for AI training, and BDCbot is one of the principal collectors for this purpose.

⚙️ Rate Limiting Policy

BDCbot is rate‑limited because its high crawl frequency can consume significant server resources, especially on smaller sites. Security teams implement threshold‑based blocking (e.g., >100 requests per minute per IP) to ensure fair resource allocation while still allowing legitimate indexing and AI data collection, as recommended by Microsoft’s own guidelines for managing crawler traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.