🤖 Overview

bsearchr&d is a web crawler operated by Baidu, Inc., specifically attributed to the company’s search engine research and development division. Its primary purpose is to collect publicly accessible web content for improving Baidu’s search indexing algorithms and for training internal AI models, such as the ERNIE family of large language models. This bot is distinct from the standard Baiduspider and is focused on experimental and R&D data gathering.

🌐 Technical Behavior

The bot initiates HTTP/1.1 GET requests with a dynamic crawl schedule that typically respects a delay of 5–10 seconds between requests, though it can burst up to 20 requests per minute during initial site discovery. It sources from IP ranges registered to Baidu in AS55967 and AS38365, primarily originating from data centers in Beijing and Shenzhen. The crawler supports both IPv4 and IPv6 connections and negotiates TLS 1.2/1.3. It fetches robots.txt before each session and parses sitemap.xml files for efficient traversal, but may occasionally ignore Allow directives for subdirectories.

📋 robots.txt Compliance

According to Baidu’s official documentation published at help.baidu.com, all Baidu-operated crawlers, including bsearchr&d, are designed to honor Disallow rules specified in robots.txt. However, independent audits (e.g., by BotScout in 2022) have noted that the bot occasionally makes requests to blocked paths when crawling from non-standard IP ranges, suggesting a configuration bug rather than intentional violation. Baidu has since patched this in crawler version 2.1.

🔍 Detection Indicators

The bot identifies itself with the User-Agent string Mozilla/5.0 (compatible; bsearchr&d/2.0; +http://www.baidu.com/search/spider.html) and sometimes uses a shortened variant Baidu Search R&D. Behavioral fingerprints include a Referer header set to the target URL, an Accept-Language of zh-CN,zh;q=0.9, and a Connection header of keep-alive. The bot sends User-Agent and From headers containing the contact email [email protected].

📊 Data Usage

Collected data is primarily used to train and fine-tune Baidu’s AI models, including the ERNIE 3.0 series and internal ranking algorithms. A secondary use is for search index refresh and query log analysis to improve Baidu’s Chinese-language search relevance. The data is stored in Baidu’s cloud infrastructure and is not shared with third parties except as required by Chinese law.

⚙️ Rate Limiting Policy

This bot is rate-limited because its R&D focus can lead to unpredictable crawl volumes, particularly after model training updates. Threshold-based blocking is enforced to prevent server overload while still allowing legitimate research data collection; typical limits are 50 requests per minute per IP, after which a 429 status is returned.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.