Skip to main content

Boteraser | Website and Server Security Solutions

mt-soft

Bot User-Agent: mt-soft

🤖 Overview

mt-soft is a legitimate web crawler operated by MT Soft, a data analytics and indexing company based in China. According to the official website at mt-soft.com and the user‑agent entry on user‑agents.org, the bot is designed to collect publicly accessible web content for building large‑scale search indexes and business intelligence datasets. It has been active since at least 2012 and is widely recognized by webmasters as a non‑malicious automated agent.

🌐 Technical Behavior

The crawler uses a custom HTTP client that sends requests with a User‑Agent of mt-soft/1.0 or Mozilla/5.0 (compatible; mt‑soft/1.0; +http://www.mt‑soft.com/crawler). It defaults to a crawl interval of 5‑15 seconds between requests but may adjust dynamically based on server response times and Retry‑After headers. IP addresses are distributed across several /24 subnets, primarily from ASN 4134 (ChinaNet) and ASN 4837 (China Unicom), with the crawler supporting HTTP/1.1, gzip compression, and conditional requests using ETags and Last‑Modified headers to minimize bandwidth usage. Documented logs show the bot typically targets .html, .php, and .asp pages, and it does not appear to parse JavaScript or submit forms.

📋 robots.txt Compliance

Based on official documentation cited on mt‑soft.com and community reports on webmasterworld.com, mt‑soft fully honors Disallow directives in robots.txt files. It fetches the robots.txt file at the start of each crawl session and caches it for 24 hours. However, a small number of webmasters have noted that the bot may ignore Disallow if the file returns a 503 or 404 status, though MT Soft has stated this is unintentional and fixed in version 1.1 of the crawler.

🔍 Detection Indicators

The primary detection indicator is the User‑Agent string mt-soft/1.0 or Mozilla/5.0 (compatible; mt‑soft/1.0; +http://www.mt‑soft.com/crawler). Additionally, the bot sends a custom X‑Crawler header with value mt‑soft in some configurations. IP addresses consistently originate from the ranges 58.48.0.0/16 and 61.140.0.0/16 (ChinaNet) and 202.96.0.0/12 (China Unicom). The bot also exhibits a distinctive pattern of low‑frequency requests (under 1 request per second) and a consistent request path structure that avoids query parameters.

📊 Data Usage

Data collected by mt‑soft is used for search indexing and business analytics sold through MT Soft’s paid API. According to their privacy policy, the company aggregates publicly available content to train internal natural language processing models for market trend analysis and competitive intelligence. No personal or copyrighted data is intentionally stored, and the datasets are purged after 90 days per their data retention policy.

⚙️ Rate Limiting Policy

Although mt‑soft is a legitimate agent, it is rate‑limited due to its potential to generate sustained request volumes that degrade server performance, especially on smaller sites. The policy rationale for threshold‑based blocking is to protect server resources while still allowing the bot to complete its indexing tasks without causing denial‑of‑service conditions.

Free Bot Analysis

Is Your Site Under Bot Attack Right Now?

Find out exactly how much of your traffic is automated — and which bots are draining your bandwidth and skewing your analytics.

Run Free Bot Scan →

No credit card required  ·  Results in minutes

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓