🤖 Overview

ERNIEBot is the AI-oriented web crawler associated with Baidu's ERNIE (Enhanced Representation through Knowledge Integration) large language model family, the engine behind the ERNIE Bot (文心一言 / Yiyan) assistant launched in March 2023 and documented at https://yiyan.baidu.com. Operated by Baidu Inc. from China, the crawler feeds content pipelines that support model pre-training and alignment, retrieval-augmented answers in Baidu Search, and the open-weight ERNIE 4.5 releases published in 2025 under the PaddlePaddle/ERNIE repository on GitHub. Baidu positions ERNIEBot alongside the long-running Baiduspider family described at https://www.baidu.com/search/spider.html.

🌐 Technical Behavior

ERNIEBot issues HTTP/1.1 and HTTP/2 GET requests that retrieve HTML first and then selectively fetch linked assets, CSS, and JavaScript needed for render-based indexing. Baidu's spider infrastructure publishes verified IPv4 ranges used by its crawlers — historically blocks such as 180.76.0.0/16, 220.181.108.0/24 and 123.125.71.0/24 — with reverse DNS resolving to hostnames in the crawl.baidu.com and baidu.com domains, the standard verification method recommended on Baidu's spider page. Crawl arrival is frequently observed in bursts rather than evenly spaced requests, and Baidu's resource platform at https://ziyuan.baidu.com exposes crawl-frequency controls to site owners who verify domain ownership. Requests typically carry Accept-Encoding: gzip, deflate, a Chinese or global egress IP, and no referrer. Baidu's documentation notes that its spiders may re-crawl changed pages within minutes for high-authority hosts.

📋 robots.txt Compliance

Baidu states that its spiders honor robots.txt directives served at the site root, following the conventions later standardized in RFC 9309; unavailability of the file (HTTP 5xx) is historically treated as permission to crawl rather than as a full disallow. Wildcard and end-anchor support has been partial and version-dependent, so operators often combine robots.txt with meta robots tags and Baidu's own crawl-frequency settings.

🔍 Detection Indicators

Requests identify themselves with an ERNIEBot token in the User-Agent, typically in the form Mozilla/5.0 (compatible; ERNIEBot/1.0; +http://www.baidu.com/search/spider.html), optionally with a legacy Baiduspider/2.0 token retained for compatibility. Because User-Agent strings are trivially spoofed, the reliable fingerprints are forward-confirmed reverse DNS to baidu.com plus membership of Baidu's published IP ranges.

📊 Data Usage

Collected documents supply training and evaluation corpora for ERNIE model generations, grounding material for Baidu Search's AI summaries, and retrieval indexes supporting the Yiyan assistant. Baidu has also released ERNIE 4.5 weights openly, meaning crawled public text can influence downstream models that third parties self-host.

⚙️ Rate Limiting Policy

ERNIEBot is treated as a rate-limited, legitimate agent: threshold-based throttling protects origin capacity during burst crawling without blocking the bot outright, preserving search visibility while keeping latency budgets for human users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.