🤖 Overview

ChatGLM-Spider is a legitimate web crawler operated by Zhipu AI (Beijing Zhipu Huazhang Technology Co., Ltd.), a Chinese AI research company founded in 2019. Its primary purpose is to collect publicly available web content—including text from articles, forums, and documentation—to train and improve the ChatGLM series of bilingual (Chinese-English) large language models, including ChatGLM-4, ChatGLM-3, and earlier versions. The crawler is explicitly documented on Zhipu AI’s official website (zhipu.ai) and in their open-source model repositories on GitHub (github.com/THUDM). It was first identified in mid-2023, around the same time ChatGLM-3 was released.

🌐 Technical Behavior

ChatGLM-Spider initiates HTTP(S) GET requests to fetch web content, typically at a moderate to high frequency—observed rates range from 1–5 requests per second per IP, depending on the target’s response speed. It uses a pool of IP addresses, predominantly from Chinese cloud providers such as Alibaba Cloud, Tencent Cloud, and occasionally from AWS China regions. The crawler follows internal rate‑limiting mechanisms, but when multiple instances run concurrently (e.g., for large-scale data collection), request volumes can spike, sometimes triggering web server defenses. It supports both HTTP/1.1 and HTTP/2 protocols and includes a "Accept-Language" header favoring Chinese (zh‑CN, zh) followed by English (en‑US). The crawler does not appear to execute JavaScript—it only retrieves static HTML—and it issues requests with a default timeout of 30 seconds.

📋 robots.txt Compliance

According to Zhipu AI’s official documentation (available at zhipu.ai/robots), ChatGLM-Spider is designed to honor robots.txt Disallow directives. However, independent webmaster reports (e.g., on community forums) note that the crawler may occasionally ignore rate-limiting instructions if the robots.txt file is not explicitly configured with "Crawl‑Delay". Zhipu AI advises website operators to include a "User‑agent: ChatGLM‑Spider" block in their robots.txt if they wish to fully exclude content. The company maintains an official list of its crawler’s behavior guidelines in a blog post titled “关于ChatGLM‑Spider爬虫的说明” (Explanation of the ChatGLM‑Spider Crawler), where they commit to respecting standard exclusion protocols.

🔍 Detection Indicators

The primary detection indicator is the User‑Agent string: "Mozilla/5.0 (compatible; ChatGLM‑Spider/1.0; +https://zhipu.ai/spider)". A variant with "ChatGLM‑Spider/2.0" was observed in late 2024. Additional identifying headers include "From: [email protected]" (sometimes) and a non‑standard "X‑Fetcher: ChatGLM‑Spider". The crawler’s IP addresses reverse‑resolve to hostnames containing "zhipu" or "glm" (e.g., glm‑crawler‑x.aliyun.zhipu.com). Behavioral fingerprints: requests arrive sequentially for linked pages (e.g., pagination links) without random intervals, and no JavaScript or cookie handling is performed.

📊 Data Usage

All content collected by ChatGLM-Spider is used exclusively for training and fine‑tuning Zhipu AI’s large language models, including ChatGLM‑2, ChatGLM‑3, and ChatGLM‑4. The data is processed through deduplication, filtering of personally identifiable information (PII), and quality scoring before being integrated into the model’s training corpus. Zhipu AI states that the crawler does not store copyrighted content verbatim; instead, extracted text is used for statistical language modeling. The company publishes a transparency report on its website detailing data sources, though it does not currently offer an opt‑out mechanism beyond robots.txt.

⚙️ Rate Limiting Policy

ChatGLM-Spider is rate‑limited because its concurrent multi‑threaded crawling can saturate server resources, especially on smaller websites. Zhipu AI recommends a default rate limit of 20 requests per minute per IP; administrators are advised to implement threshold‑based blocking (e.g., 5 req/s for 30 seconds) to prevent excessive load without permanently banning the legitimate crawler. This policy balances data collection needs with fair use of server capacity.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.