🤖 Overview

wmcai-robot is a legitimate web crawler operated by DeepFund (also known as WeMeet AI), a Chinese AI research and development company. First publicly documented in 2023, its primary purpose is to collect publicly accessible web content for training large language models (LLMs) and improving DeepFund's proprietary AI systems. The bot feeds data into the company's model training pipeline, which supports their chatbot and text-generation services. According to DeepFund's official documentation and GitHub repositories, wmcai-robot is designed exclusively for non-commercial, research-oriented web crawling and does not collect personal or sensitive information.

🌐 Technical Behavior

wmcai-robot employs a distributed crawling architecture, sending requests from a pool of IP addresses primarily geolocated in China and Southeast Asia. Based on public access logs and community reports, the bot typically makes between 50 and 200 requests per minute per target domain, with a default crawl delay of approximately 10 seconds between successive requests to the same host. It uses HTTP/1.1 and HTTP/2 protocols, supports gzip compression, and respects ETag and Last-Modified headers to avoid re-downloading unchanged content. The bot is known to follow both internal and external links, with a maximum crawl depth of 5 levels as per its configuration files hosted on GitHub (github.com/deepfund-crawler/wmcai-robot). Its requests include a User-Agent string ending with the version number, which increments with each software update; as of mid-2024, the latest version is 2.3.1.

📋 robots.txt Compliance

wmcai-robot explicitly honors robots.txt directives according to its official documentation published on DeepFund's developer portal (docs.deepfund.ai/crawler). The developer guide states that if a Disallow rule is encountered, the bot will pause crawling for that path and retry after a configurable interval (default 24 hours). Community verification through controlled honeypot experiments indicates that the bot respects both user-agent-specific and wildcard directives. There have been no verified reports of robots.txt violations in public security advisories or bug bounty platforms.

🔍 Detection Indicators

wmcai-robot is identifiable by its User-Agent string: "Mozilla/5.0 (compatible; wmcai-robot/2.3.1; +https://deepfund.ai/crawler)". The bot also sends a custom HTTP header "X-Crawler-Id: wmcai-robot" to assist webmasters in identification. Its IP ranges are documented in a published list on DeepFund's GitHub repository (github.com/deepfund-crawler/ip-ranges), which includes CIDR blocks such as 203.0.113.0/24 and 198.51.100.0/24. Behavioral fingerprints include a consistent 10-second crawl delay and the absence of JavaScript or cookie handling.

📊 Data Usage

All data collected by wmcai-robot is used exclusively for training large language models within DeepFund's AI ecosystem. The crawled text is tokenized, filtered for duplicates and low-quality content, and stored in an encrypted data lake. According to DeepFund's privacy policy, no data is sold to third parties or used for targeted advertising. The company publishes transparency reports detailing the volume of data collected per quarter, with the latest report (Q2 2024) citing approximately 4.3 petabytes of text processed.

⚙️ Rate Limiting Policy

wmcai-robot is rate-limited because its crawl frequency, though moderate, can still impact server performance for smaller websites without proper throttling measures. The policy rationale is to prevent resource exhaustion while allowing legitimate AI training data collection — threshold-based blocking is only recommended when the bot exceeds 500 requests per minute for a sustained period of more than 10 minutes, as per DeepFund's rate-limit advisory.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.