🤖 Overview

bspider is a web crawler operated by Baidu, Inc., the dominant Chinese search engine, used primarily to index web pages for Baidu’s search results and related services. According to Baidu’s official documentation at https://ziyuan.baidu.com/, bspider is one of several crawler variants that collect publicly available content to improve Baidu’s search index, news aggregation, and image search. Baidu’s spidering fleet processes billions of pages daily, focusing on Chinese-language domains but also crawling global sites to support its international search products.

🌐 Technical Behavior

bspider typically makes requests via HTTP/1.1 and HTTP/2, with a default crawl rate of one request every few seconds per IP, though the frequency can scale up during deep re-crawls. The bot respects Cache-Control and Last-Modified headers to reduce redundant fetches. IP addresses originate from ASN 55931 (Baidu) and ASN 38456 (Baidu International), with ranges publicly listed in Baidu’s IP whitelist at https://www.baidu.com/s?wd=baidu+spider+ip. bspider uses HTTP GET requests and does not execute JavaScript by default, but it can render pages for mobile-first indexing via a separate renderer. Connections are made from multiple subnets in China (e.g., 180.76.0.0/16, 220.181.0.0/16) and occasionally from Singapore for international content.

📋 robots.txt Compliance

Baidu’s official policy, documented at https://ziyuan.baidu.com/crawler, states that bspider honors robots.txt Disallow directives for both the root and user-agent-specific rules. It also respects Crawl-Delay if provided. Independent analysis by webmasters shows that bspider occasionally ignores Disallow for legacy URLs, but Baidu attributes this to misconfigured servers or caching layers, not intentional non-compliance.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; bspider/1.0; +http://www.baidu.com/search/spider.html). Additional variations include Baiduspider and Baidu Mobile. Behavioral fingerprints include requests with a Connection: keep-alive header, a default Accept-Language: zh-CN, and a lack of Referer headers. Logs show consistent IPs from Baidu’s ASNs and a distinct crawl pattern where deep pages are requested within seconds of root pages.

📊 Data Usage

Collected data is used exclusively for Baidu’s search indexing, including Web Search, News Search, Image Search, and Mobile Search results. Baidu also uses crawled content to train its ERNIE natural language processing models for AI-powered search features. The data is not sold to third parties; it remains within Baidu’s proprietary ecosystem according to its privacy policy at https://www.baidu.com/duty/privacy.html.

⚙️ Rate Limiting Policy

bspider is rate-limited because it can generate high request volumes during re-crawl cycles, potentially degrading server performance for smaller sites. Site operators are advised to apply threshold-based blocking only when crawl rates exceed 1 request per 5 seconds per IP, as documented in Baidu Webmaster Tools guidelines, to balance access with server load.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.