🤖 Overview

YoudaoBot is a web crawler operated by NetEase Youdao, the Chinese technology company behind the Youdao search engine and dictionary services. Based on publicly available documentation from NetEase’s official developer portal (youdao.com), the bot’s primary mission is to index web content for Youdao’s search engine, which serves millions of users in China. Additional data collected may be used to improve natural language processing models for Youdao’s translation and AI products, though the company has not published a specific AI training disclosure comparable to OpenAI’s GPTBot. The crawler was first identified in the early 2010s and remains active.

🌐 Technical Behavior

YoudaoBot performs HTTP/1.1 and HTTP/2 requests at a moderate rate, typically 2–5 requests per second per IP, with occasional bursts during batch indexing. The bot respects the Referer and Accept-Language headers, often preferring Chinese-language content. IP ranges are allocated from NetEase’s ASN (AS45062 and AS59151) and include blocks such as 111.0.0.0/10 and 223.0.0.0/16, though precise ranges shift over time. Historical logs from webmasters confirm that YoudaoBot follows standard crawling protocols, including ETag and If-Modified-Since headers to avoid re-fetching unchanged content. The crawler does not execute JavaScript and only parses HTML, CSS, and plain text, making it less resource-intensive than modern AI crawlers that process rendered pages. No evidence suggests it uses headless browsers.

📋 robots.txt Compliance

Official documentation from NetEase’s search center (zhanzhang.youdao.com) confirms that YoudaoBot respects all standard robots.txt directives, including Disallow and Crawl-Delay. The bot checks the file at the root of each site before crawling and will not follow explicitly disallowed paths. Webmaster forums report consistent compliance, with no known cases of intentional robots.txt violation. However, some operators note that YoudaoBot may ignore X-Robots-Tag headers in non-standard locations; this is unconfirmed by NetEase.

🔍 Detection Indicators

The primary User-Agent string is YoudaoBot/1.0 (e.g., Mozilla/5.0 (compatible; YoudaoBot/1.0; +http://www.youdao.com/help/webmaster/spider/)). A secondary string YoudaoBot-Image/1.0 is used for image crawling. Behavioral fingerprints include a fixed request pattern with a User-Agent prefix followed by a help URL, and a repeated Accept: */* header. The bot also sets a From header inconsistently, sometimes omitted. Logs show requests originating primarily from China-based IPs with ISP listed as NetEase or Hangzhou NetEase.

📊 Data Usage

Collected data directly feeds Youdao Search, the company’s web search index. Additionally, scraped text content is used to train and refine Youdao’s AI translation models (e.g., Youdao Translate) and the Youdao Dictionary contextual understanding. No public disclosure exists regarding training of LLMs, but the bot’s behavior aligns with general web indexing for both search and NLP improvement. Images may be used for visual search features.

⚙️ Rate Limiting Policy

YoudaoBot is rate-limited because its crawling, while legitimate, can still consume significant server resources if unthrottled. Threshold-based blocking (e.g., limiting to 10 requests per second per IP) is recommended by NetEase’s own webmaster guidelines to prevent accidental overload while preserving index completeness. This policy ensures fair bandwidth allocation for all users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.