QwenBot
Bot User-Agent:qwenbot
🤖 Overview
QwenBot is a web crawler operated by Alibaba Group for the Qwen large language model family and Alibaba Cloud AI services. It discovers publicly accessible web content used to train and improve Qwen models, including Qwen Chat and related retrieval features. Alibaba documents QwenBot as a legitimate automated agent at qwen.ai/qwenbot, not a user-driven fetcher.
🌐 Technical Behavior
QwenBot issues HTTP and HTTPS GET requests for HTML, PDF, plain text, and structured files such as JSON and XML. It typically sends Accept: text/html,application/xhtml+xml and Accept-Encoding: gzip, deflate, br, and it does not execute JavaScript or render client-side content. Telemetry from independent bot trackers places it on Alibaba Cloud AS45102 and AS37963, with distributed source IPs that change over time. It follows links and sitemaps, revisits pages for updates, and parallelizes fetches across worker nodes, so request rates can become high. It supports HTTP/1.1 and HTTP/2 and normally sends no cookies or authenticated session headers.
📋 robots.txt Compliance
Alibaba's QwenBot documentation states that the crawler honors robots.txt directives, including Disallow rules and crawl-delay preferences where supported. Webmaster reports generally confirm QwenBot stops requesting paths explicitly disallowed for its user agent, though updated robots.txt files can take hours to propagate through distributed caching. Site owners can apply targeted rules using the token QwenBot.
🔍 Detection Indicators
The main indicator is the User-Agent string, commonly seen as QwenBot/1.0 with a reference URL such as + Similar Threats Free Bot Analysis Find out exactly how much of your traffic is automated — and which bots are draining your bandwidth and skewing your analytics. No credit card required · Results in minutes ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.Is Your Site Under Bot Attack Right Now?
