🤖 Overview

YiyanBot is a web crawler operated by Baidu, Inc., the Chinese multinational technology company, and is the primary data collection agent for Baidu’s ERNIE Bot (文心一言) — a large language model (LLM) first launched publicly in March 2023. According to Baidu’s official developer documentation (ai.baidu.com), YiyanBot is designed to crawl publicly accessible web pages to gather high-quality, diverse Chinese-language content for training and improving the ERNIE series of generative AI models. The bot was first documented in Baidu’s search engine guidelines in early 2024, following China’s regulatory push for transparent AI data sourcing.

🌐 Technical Behavior

YiyanBot employs a headless Chromium-based crawling engine, mimicking a modern desktop browser to render JavaScript-heavy pages. It sends requests at a controlled rate of approximately 5–10 requests per second per IP, as per Baidu’s published crawl rate guidelines (ziyuan.baidu.com). The bot’s IP ranges are part of Baidu’s ASN (AS55967 and AS38365), with public prefixes like 180.76.0.0/16 and 220.181.0.0/16. It uses standard HTTPS/1.1 and HTTP/2 protocols and sends an Accept-Language: zh-CN,zh;q=0.9 header alongside a Referer header that often points to the originating search query. YiyanBot does not crawl URLs with authentication or login requirements but will access robots.txt before any page fetch. Crawl frequency is dynamically adjusted based on server load signals (e.g., 429 responses).

📋 robots.txt Compliance

Baidu explicitly states that YiyanBot honors the Disallow directives in robots.txt, as outlined in their official search engine documentation (ziyuan.baidu.com/college/courseinfo?id=300). However, the bot does not respect Crawl-Delay directives in robots.txt; rate limiting is instead managed by Baidu’s own crawl scheduler. Webmasters can block YiyanBot entirely by adding User-agent: YiyanBot followed by Disallow: /. This policy is consistent with Baidu’s approach for all its crawlers (Baiduspider, Baidu-YiyanBot).

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; YiyanBot/1.0; +https://ai.baidu.com/ai-doc/REFERENCE/Ck3dwjhh3). A secondary string occasionally observed is Baidu-YiyanBot/1.0. Behavioral fingerprints include a consistent Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 header and a From header containing [email protected]. The robot’s IP reverse DNS typically resolves to *.baiduspider.com or *.baidu.com. Web administrators can verify the bot’s legitimacy via Baidu’s IP reverse lookup tool (ziyuan.baidu.com/spider/verify).

📊 Data Usage

Data collected by YiyanBot is used exclusively for training and fine-tuning Baidu’s ERNIE Bot (also known as 文心一言) and related AI products. The crawled content includes web pages, blog posts, forum threads, and news articles — primarily in Simplified Chinese. Baidu’s AI Data Usage Policy (ai.baidu.com/legal) states that personal identifiable information is stripped before training, and the data is stored in Baidu’s internal secure data lakes. The collected data also improves Baidu Search’s AI-generated answer summaries.

⚙️ Rate Limiting Policy

YiyanBot is rate-limited by default to prevent overwhelming small servers, but its dynamic scheduler can increase frequency on high-capacity hosts. Web application owners are advised to set a reasonable threshold (e.g., 20 requests per minute per IP) and return a 429 Too Many Requests response if the bot exceeds acceptable load, as Baidu’s crawler will respect that status code and back off.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.