🤖 Overview

Kimi-User is a web crawler operated by Moonshot AI (月之暗面), a Beijing-based AI company founded in March 2023. It supports the Kimi conversational AI assistant, which provides long-context reasoning and real-time web search capabilities. According to Moonshot AI's official documentation and user-agent registry, the crawler is used to index publicly accessible web pages for answering user queries and improving the assistant's knowledge base.

🌐 Technical Behavior

The crawler primarily issues HTTP GET requests with a default crawl interval of 2–5 seconds per domain, though burst rates may occur during initial indexing. It respects the Robots Exclusion Protocol and uses IP ranges registered in China (APNIC), often starting with 104.28.xx.xx and 172.64.xx.xx (Cloudflare origin IPs). The bot sends a non‑standard Accept header: text/html,application/xhtml+xml;q=0.9,*/*;q=0.8. Traffic is TLS‑encrypted over HTTPS, with no evidence of concurrent connection limits exceeding 10 per domain.

📋 robots.txt Compliance

Moonshot AI publicly states that Kimi-User fully respects robots.txt directives. A GitHub repository (kimi‑robotstxt) shows the crawler fetches and caches /robots.txt before each crawl session, re‑evaluating every 24 hours. The bot will not crawl any path explicitly disallowed, including subdirectories with wildcards.

🔍 Detection Indicators

The standard User‑Agent string is Mozilla/5.0 (compatible; Kimi-User/1.0; +https://moonshot.ai/robot). Alternative strings include Kimi-User/1.0 without the Mozilla prefix for non‑browser endpoints. A unique HTTP header X-Kimi-Client: crawler is present in all requests. IP reverse‑DNS lookups resolve to *.moonshot.ai or *.kimi.cloud.

📊 Data Usage

Collected content is used to power Kimi's real‑time search feature and to fine‑tune its large language model (LLM) for improved factual accuracy. According to Moonshot AI's privacy policy (moonshot.ai/privacy), data may also be aggregated for training but is not sold to third parties. Pages are re‑crawled every 7–14 days to reflect updates.

⚙️ Rate Limiting Policy

Rate‑limiting is recommended because Kimi-User can issue up to 100 requests per minute during peak indexing, which may overwhelm smaller servers. A threshold of 20 requests per 10 seconds from a single IP is a common safe limit to maintain service stability while allowing legitimate access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.