mt-soft
mt-soft is a legitimate web crawler operated by MT Soft, a data analytics and indexing company based in China. According to the official website at mt-soft.com and the user‑agent entry on user‑agents.org, the bot is designed to collect publicly accessible web content for building large‑scale search indexes and business intelligence datasets. It has been active since at least 2012 and is widely recognized by webmasters as a non‑malicious automated agent.
The crawler uses a custom HTTP client that sends requests with a User‑Agent of mt-soft/1.0 or Mozilla/5.0 (compatible; mt‑soft/1.0; +http://www.mt‑soft.com/crawler). It defaults to a crawl interval of 5‑15 seconds between requests but may adjust dynamically based on server response times and Retry‑After headers. IP addresses are distributed across several /24 subnets, primarily from ASN 4134 (ChinaNet) and ASN 4837 (China Unicom), with the crawler supporting HTTP/1.1, gzip compression, and conditional requests using ETags and Last‑Modified headers to minimize bandwidth usage. Documented logs show the bot typically targets .html, .php, and .asp pages, and it does not appear to parse JavaScript or submit forms.
Based on official documentation cited on mt‑soft.com and community reports on webmasterworld.com, mt‑soft fully honors Disallow directives in robots.txt files. It fetches the robots.txt file at the start of each crawl session and caches it for 24 hours. However, a small number of webmasters have noted that the bot may ignore Disallow if the file returns a 503 or 404 status, though MT Soft has stated this is unintentional and fixed in version 1.1 of the crawler.
The primary detection indicator is the User‑Agent string mt-soft/1.0 or Mozilla/5.0 (compatible; mt‑soft/1.0; +http://www.mt‑soft.com/crawler). Additionally, the bot sends a custom X‑Crawler header with value mt‑soft in some configurations. IP addresses consistently originate from the ranges 58.48.0.0/16 and 61.140.0.0/16 (ChinaNet) and 202.96.0.0/12 (China Unicom). The bot also exhibits a distinctive pattern of low‑frequency requests (under 1 request per second) and a consistent request path structure that avoids query parameters.
Data collected by mt‑soft is used for search indexing and business analytics sold through MT Soft’s paid API. According to their privacy policy, the company aggregates publicly available content to train internal natural language processing models for market trend analysis and competitive intelligence. No personal or copyrighted data is intentionally stored, and the datasets are purged after 90 days per their data retention policy.
Although mt‑soft is a legitimate agent, it is rate‑limited due to its potential to generate sustained request volumes that degrade server performance, especially on smaller sites. The policy rationale for threshold‑based blocking is to protect server resources while still allowing the bot to complete its indexing tasks without causing denial‑of‑service conditions.
Similar Threats
Free Bot Analysis
Find out exactly how much of your traffic is automated — and which bots are draining your bandwidth and skewing your analytics.
Run Free Bot Scan →No credit card required · Results in minutes
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.