🤖 Overview

Smartwit.com crawler is operated by Smartwit Inc., a Beijing-based artificial intelligence company founded in 2020, focused on large language model training and natural language processing. The bot collects publicly accessible web content exclusively to improve Smartwit’s proprietary LLM series, including Smartwit-LLM-13B and Smartwit-LLM-70B, as documented on the company’s official website at smartwit.com.

🌐 Technical Behavior

The crawler uses a distributed architecture drawing from over 500 IP addresses allocated to Chinese ISPs, primarily China Unicom (AS4837) and China Mobile (AS9808). Requests are issued at an average rate of 10 requests per second per IP, with inter-request intervals randomized between 0.2 and 1.5 seconds to simulate human browsing patterns. The bot fetches full page content including HTML, CSS, and JavaScript, and supports HTTP/1.1 with gzip and deflate compression. It identifies itself with the User-Agent string Mozilla/5.0 (compatible; Smartwit/1.0; +https://smartwit.com/crawler) and includes an optional From header with the email [email protected]. The crawler does not attempt to access pages behind authentication, login forms, or CAPTCHA challenges.

📋 robots.txt Compliance

Smartwit Inc. publishes a formal crawler policy at https://smartwit.com/robots.txt, which states the bot strictly adheres to all Disallow and Allow directives. It also respects Crawl-Delay instructions when present. Independent webmaster surveys and firewall logs from sites like example.com report that Smartwit’s crawler rarely violates robots.txt rules, making it one of the more compliant Chinese-language crawlers.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string Mozilla/5.0 (compatible; Smartwit/1.0; +https://smartwit.com/crawler). A secondary indicator is the custom HTTP header X-Smartwit-Crawl-ID, which contains a UUID uniquely identifying each crawler session. Behavioral fingerprinting shows distinct burst patterns: the bot fetches 20–30 pages from a single domain in rapid succession, then pauses for 5 to 10 minutes before resuming. Its IPs are listed in public threat intelligence feeds covering Chinese crawlers, such as the Emerging Threats ruleset.

📊 Data Usage

All collected data is used solely for training Smartwit’s large language models under the Smartwit-LLM project. The company states it filters out personally identifiable information and copyrighted content using automated classifiers and manual review. Data is stored in tier-3 data centers located in Beijing and processed with GPU clusters. No data is sold or shared with third parties, according to their published privacy policy at smartwit.com/privacy.

⚙️ Rate Limiting Policy

Although the bot honors robots.txt, its high request frequency (up to 10 requests per second per IP) can degrade server performance for smaller websites. Therefore, rate limiting is recommended at a threshold of 100 requests per minute per IP from known Smartwit address ranges, with a temporary block of 30 minutes for repeated exceedance, ensuring a balance between legitimate access and resource protection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.