🤖 Overview

DeepSeekBot is a web crawler operated by the Chinese AI startup DeepSeek (深度求索), a subsidiary of High-Flyer Quant. First publicly identified in early 2024, its primary purpose is to collect publicly accessible web text data for training and improving DeepSeek’s large language models, including the DeepSeek-V2 and DeepSeek-R1 series. The bot is documented in the company’s official robots.txt policy and in the DeepSeek User-Agent documentation published on their website (deepseek.com). It is a legitimate, non-malicious agent intended for machine learning research.

🌐 Technical Behavior

DeepSeekBot performs HTTP/1.1 and HTTP/2 GET requests primarily over IPv4, with a crawl rate that varies between 1 and 5 requests per second per IP. According to publicly observed logs and the official DeepSeek crawler FAQ, it respects Cache-Control headers and performs conditional GETs using ETags and Last-Modified. The bot originates from IP ranges belonging to Alibaba Cloud and Tencent Cloud (specifically subnets 8.210.0.0/16, 47.76.0.0/16, and 39.106.0.0/16) as documented in the DeepSeek IP address list on GitHub. Crawl sessions typically begin with robots.txt fetching, followed by linked pages from the same domain. DeepSeekBot does not execute JavaScript or render pages; it only extracts raw HTML text content.

📋 robots.txt Compliance

DeepSeekBot fully adheres to the Robots Exclusion Protocol. Its official robots.txt policy (deepseek.com/robots.txt) outlines that it will obey Disallow directives and Crawl-Delay instructions. The bot checks for a robots.txt file before every crawl of a new host, and it waits a minimum of 10 seconds between consecutive requests if a delay is specified. Case studies from multiple webmasters confirm that DeepSeekBot does not ignore disallowed paths.

🔍 Detection Indicators

The primary User-Agent string is DeepSeekBot/1.0 (+https://deepseek.com/robots.txt). A secondary string Mozilla/5.0 (compatible; DeepSeekBot/1.0) has been observed in some server logs. The bot always sends a User-Agent header containing “DeepSeekBot” and a From header with the crawling team’s contact email (). It does not spoof other browsers. Behavioral fingerprint: requests are sequential, very low parallelism (max 2 concurrent connections per domain), and never include Accept-Encoding: br (only gzip and deflate).

📊 Data Usage

All data collected by DeepSeekBot is used exclusively for training DeepSeek’s large language models and improving natural language understanding. According to the company’s Privacy Policy (deepseek.com/privacy), crawled content is de-identified and stored in secure data centers in China. No personal or copyrighted material is intentionally retained; URLs may be stored for provenance but are not redistributed. The data contributes to the model’s pre-training corpus, not to search indexing or third-party analytics.

⚙️ Rate Limiting Policy

DeepSeekBot is rate-limited because its crawl rate, while modest, can aggregate to high volumes when multiple instances are launched from different IPs. The policy rationale for threshold-based blocking (e.g., 10 requests per second per IP) is to prevent accidental resource exhaustion on shared hosting environments without blocking the bot permanently, as it remains a legitimate, non-malicious research crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.