🤖 Overview

wenbin is a web crawler operated by Wenbin Technology Co., Ltd., a Chinese AI research firm, first documented in early 2024. Its primary purpose is to collect publicly available web content for training large language models used in their conversational AI platform. The bot was identified through official announcements on the company’s website (wenbin-ai.com) and a GitHub repository (github.com/wenbin-ai/crawler) that details its crawling guidelines.

🌐 Technical Behavior

The wenbin crawler uses a distributed architecture with IP ranges primarily originating from Chinese ASNs, including AS4134 (ChinaNet) and AS4837 (China Unicom). It sends HTTP/1.1 requests with a default interval of 2 seconds between page fetches, but can be configured to burst up to 5 requests per second during peak indexing. The crawler respects ETag and Last-Modified headers to avoid re-downloading unchanged content, and it fetches both HTML and JavaScript-rendered pages using a headless Chromium instance. According to the official documentation, it follows all a and link tags, but does not process images or media files. Requests are sent with a custom Accept-Language header set to zh-CN,en;q=0.9 and a default timeout of 30 seconds.

📋 robots.txt Compliance

The wenbin crawler fully honors robots.txt directives as per its published policy. It reads the file at each domain’s root and will not crawl any path explicitly listed under Disallow: / for its user-agent. The official GitHub repository states that it also obeys Crawl-Delay values when set to >5 seconds. However, it does not respect noindex meta tags or X-Robots-Tag headers unless specifically configured by the site owner via a whitelist (Allow directive).

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; wenbin/1.0; +https://wenbin-ai.com/bot). Secondary variants include wenbin-crawler/1.0 and WenbinSpider/1.0. Behavioral fingerprints include a fixed request frequency of 0.5-2 seconds per domain and the absence of common browser features like cookies or session persistence beyond a single request. The crawler sends a custom header X-Wenbin-Version set to the build number, aiding identification.

📊 Data Usage

Collected data is used exclusively for training Wenbin AI’s large language models, including their flagship Wenxin‑LLM. The company’s privacy policy states that raw crawled content is anonymized and stripped of personally identifiable information before being added to training corpora. No data is sold or shared with third parties, and a deletion request process is available via their website.

⚙️ Rate Limiting Policy

The wenbin crawler is rate-limited because its distributed IP pool can generate sustained request loads that mimic DDoS activity, especially on sites with low bandwidth. A threshold-based blocking policy (e.g., exceeding 50 requests per minute from a single IP) is recommended to protect server resources while allowing the legitimate crawling to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.