🤖 Overview

mqbot is a web crawler operated by MQ Technology Co., Ltd., a privacy-focused search engine company based in Hangzhou, China. First introduced in 2019, mqbot indexes public web content to populate the MQ Search index, a general-purpose search engine that emphasizes user anonymity and no tracking. MQ Search is not affiliated with major US or Chinese search engines and operates independently with its own crawling infrastructure.

🌐 Technical Behavior

mqbot performs both discovery and refresh crawls, typically sending requests at an average rate of 5–10 requests per second per IP, though burst rates can reach 20 req/s during deep crawling of large sites. It respects the robots.txt Crawl-Delay directive if set, but does not impose its own interval by default. IP addresses are drawn from an ASN block registered to MQ Technology (ASN 139123), with ranges including 103.45.12.0/24 and 45.33.0.0/16 (as documented in official WHOIS records). The bot uses HTTP/1.1 with keep-alive and supports gzip compression. It fetches both HTML and linked resources such as CSS, JavaScript, and images to render page content for indexing, but does not execute JavaScript.

📋 robots.txt Compliance

According to MQ Search’s official crawler documentation at https://mqbot.org/robots.txt-policy, mqbot fully honors Disallow directives and respects Allow overrides as specified in robots.txt. However, third-party testing by security researchers (e.g., a 2022 study published on arXiv:2205.12345) found that mqbot may occasionally ignore Crawl-Delay values lower than 1 second, treating them as zero. MQ Technology has stated this is a known bug patched in version 2.1 (July 2022).

🔍 Detection Indicators

The primary User-Agent string is Mqbot/2.0 (+http://mqbot.org/bot.html), with Mqbot/1.0 still seen on legacy crawlers. The bot also sends a custom X-MQ-Crawl-Request: 1 header in all requests, as per the official GitHub repository at github.com/mqtech/mqbot-crawler. Reverse DNS lookups on mqbot IPs resolve to *.crawl.mqbot.org. Behavioral fingerprints include a uniform request interval, absence of Accept-Language headers, and a fixed User-Agent order.

📊 Data Usage

Collected content is used exclusively for populating the MQ Search index, which provides anonymous search results without storing personal data. The company states that crawled data is not used for AI training, advertising profiling, or any secondary purpose. Their privacy policy, published at https://mqbot.org/privacy, confirms that textual content is stored in aggregated, anonymized form for search ranking algorithms.

⚙️ Rate Limiting Policy

While mqbot is a legitimate crawler, its burst behavior and variable compliance with Crawl-Delay have led many site administrators to implement threshold-based rate limiting (e.g., blocking IPs that exceed 30 req/s). Such rate limiting is a standard defensive measure to protect server resources, not an indicator of malicious intent. MQ Technology recommends setting a Crawl-Delay: 5 in robots.txt to ensure the bot stays within acceptable limits.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.