🤖 Overview

TongyiBot is a web crawler operated by Alibaba Cloud, a subsidiary of Alibaba Group, officially announced in 2023 as part of the Tongyi Qianwen large language model ecosystem. Its primary purpose is to collect publicly accessible web content to support the training and improvement of Alibaba’s generative AI models, including Tongyi Qianwen and related enterprise AI services like Tongyi Lingma (coding assistant). The bot is documented in Alibaba’s official search engine crawler list and is distinguishable from traditional search engine crawlers by its AI-training oriented data collection scope.

🌐 Technical Behavior

TongyiBot performs both GET and HEAD requests over HTTP/1.1 and HTTP/2, targeting HTML pages, text files, and structured data (JSON-LD, microdata). Official Alibaba documentation indicates it respects a default crawl rate of 1 request per 2 seconds but may dynamically increase under low server load. IP ranges are drawn from Alibaba Cloud’s public CIDR blocks, primarily in the 47.88.0.0/14 and 8.209.0.0/16 ranges, with geolocation mostly in China, Singapore, and the US. The bot uses a CONSISTENT user-agent string (see below) and sends an x-robot-tag: TongyiBot header for identification. Crawling is done on a best-effort basis, with no persistent session affinity across requests.

📋 robots.txt Compliance

According to Alibaba’s official TongyiBot FAQ (available at help.aliyun.com), the bot fully obeys Disallow directives in robots.txt and respects Crawl-Delay instructions when present. It also supports the X-Robots-Tag header for per-URL control. Testing by third-party security researchers has confirmed that TongyiBot does not ignore robots.txt and will not crawl paths explicitly blocked.

🔍 Detection Indicators

The canonical User-Agent string is “Mozilla/5.0 (compatible; TongyiBot/1.0; +https://tongyi.aliyun.com/robot)”. A secondary variant without the Mozilla prefix has also been observed. Behavioral fingerprints include sequential crawling of URLs with minimal time variance (typically 2–3 seconds between requests) and absence of browser-like request headers (no Accept-Language or Sec-Fetch-*). The bot also includes the custom header From: [email protected].

📊 Data Usage

Collected data is used exclusively for training Alibaba’s large language models, including Tongyi Qianwen (the general-purpose model), Tongyi Lingma (code generation), and future iterations. Alibaba’s privacy policy (published at tongyi.aliyun.com/privacy) states that text content is processed to improve model accuracy, with no storage of personally identifiable information. The data is not used for advertising or third-party sharing.

⚙️ Rate Limiting Policy

TongyiBot is rate-limited by most commercial web applications due to its potentially aggressive crawling patterns under high load. The policy rationale is to prevent resource exhaustion while still allowing reasonable AI training data collection; thresholds are typically set at 100 requests per minute per IP, with temporary blocks triggered by exceeding that rate.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.