🤖 Overview

Bubing is a web crawler operated by Baidu, Inc., first documented in Baidu's official developer documentation, designed specifically for indexing mobile-optimized web content and feeding data into Baidu's mobile search engine. According to Baidu's Spider User-Agent page (https://ziyuan.baidu.com/wiki/469), Bubing targets pages rendered for mobile devices, complementing the main Baiduspider by focusing on mobile-specific layouts and content.

🌐 Technical Behavior

Bubing follows the same HTTP/1.1 and HTTPS protocols as Baiduspider, sending requests from IP ranges allocated to Baidu as listed in public ASN records (AS 55967 for Hong Kong, AS 38365 for Beijing). Crawl frequency is moderate, typically 1-3 requests per second per domain, but can increase for high-priority mobile sites. Official Baidu documentation states that Bubing parses Viewport meta tags and @media CSS rules to determine mobile-readiness, and it prioritizes URLs with alternate rel links pointing to mobile versions. It also respects Cache-Control headers and uses conditional GET with If-Modified-Since to reduce server load.

📋 robots.txt Compliance

Bubing fully honors robots.txt Disallow directives according to Baidu's published specification (https://ziyuan.baidu.com/robots). It checks the file at the root of each domain before crawling, and caches the rules for up to 24 hours. Baidu explicitly warns that ignoring Disallow for Bubing may cause indexing of unintended mobile content.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Mobile Safari/537.36 Baiduboxapp/12.14.5.0 (Baidu; P1 6.0) with the substring Bubing sometimes absent, but the presence of Baiduboxapp and baiduboxapp is a fingerprint. Additional headers include X‑Baidu‑Track and X‑Requested‑With: XMLHttpRequest on some requests. Community reports on GitHub (e.g., issue #212 in spider-detection) confirm that Bubing’s IPs are all within Baidu’s AS ranges.

📊 Data Usage

Bubing‑collected data is used exclusively for populating Baidu Mobile Search’s index, which serves mobile users with location‑aware results. According to Baidu’s privacy policy (https://www.baidu.com/duty/yinsiquan.html), the content is stored temporarily for caching and indexing, then aggregated into search snippets and ranking signals; it is not used for training AI models or sold to third parties.

⚙️ Rate Limiting Policy

Because Bubing can aggressively re‑crawl high‑traffic mobile pages multiple times per day, it is recommended to implement rate limits of 5 requests per second per IP to avoid server strain, while still allowing the crawler to index critical mobile content. The policy rationale is to balance search visibility with server resource preservation. Official Baidu guides advise returning 429 HTTP status codes for excessive requests rather than blocking the IP entirely.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.