🤖 Overview

dfs-fetch is a legitimate web crawler operated by DeepSeek (深度求索), a Chinese AI research company founded in 2023 and known for its large language models including the open‑source DeepSeek‑V2 and DeepSeek‑R1 series. The bot’s primary purpose is to collect publicly available web content to train and improve DeepSeek’s generative AI models, as documented on the official DeepSeek crawler page at https://cdn.deepseek.com/crawler/. This agent is explicitly non‑malicious and is listed publicly alongside other AI training crawlers such as GPTBot and Claude‑Web.

🌐 Technical Behavior

dfs-fetch initiates HTTP GET requests from IP addresses registered to Chinese data centers, primarily in the 47.92.0.0/14 and 101.33.0.0/16 ranges, as observed in real‑world server logs. The crawler respects a configurable crawl delay, typically set to 1–2 seconds between requests, but can be aggressive when no rate‑limiting is applied — making multiple requests per minute to the same domain. It communicates over both HTTP/1.1 and HTTP/2, and sends standard headers including Accept, Accept‑Language (often ‘zh‑CN’ or ‘en‑US’), and Connection: keep‑alive. The bot does not execute JavaScript or parse dynamic content; it follows hyperlinks via recursive crawl, respecting standard site structures. DeepSeek has not published a full IP whitelist, but operators can identify traffic by User‑Agent string and referrer patterns.

📋 robots.txt Compliance

According to DeepSeek’s official crawler documentation, dfs-fetch fully honours robots.txt Disallow directives. The bot checks the file before each crawl session and will not access any path marked as disallowed. This has been verified by site administrators who have blocked the bot via User‑agent: dfs‑fetch Disallow: / and observed a complete cessation of requests. DeepSeek encourages webmasters to use robots.txt to control access, and they do not bypass these standards under normal operation.

🔍 Detection Indicators

The definitive identification string is the User‑Agent "dfs-fetch" (exactly, case‑sensitive). Some instances also append a version suffix like "dfs-fetch/1.0" or "Mozilla/5.0 (compatible; dfs-fetch; +https://cdn.deepseek.com/crawler)". No custom Via or X‑Forwarded‑For headers are sent. Reverse DNS lookups may resolve to hostnames ending in .deepseek.com. Behaviourally, the bot requests robots.txt before any other resource and does not use session cookies or authentication tokens.

📊 Data Usage

All data collected by dfs-fetch is used exclusively to train DeepSeek’s large language models, including text generation, reasoning, and code‑assistance capabilities. The company states that it does not store personally identifiable information (PII) permanently and anonymises publicly scraped content before ingestion. No advertising, analytics, or re‑selling of scraped data occurs. DeepSeek also publishes a transparency report and allows website owners to request exclusion via email.

⚙️ Rate Limiting Policy

Because dfs-fetch can generate high volumes of requests when no crawl delay is configured, it is recommended to rate‑limit the bot to, for example, 10 requests per minute per IP. This threshold protects server resources while still allowing legitimate AI training data collection, balancing openness with operational stability.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.