pangubot
PanguBot is a web crawler operated by Huawei Cloud, part of the Pangu series of large language models developed by Huawei’s Noah’s Ark Lab. First publicly documented in late 2023, its purpose is to collect publicly accessible web text for training and improving the Pangu NLP models, which compete with GPT and LLaMA variants. The bot feeds data into Huawei’s AI training infrastructure, including the Pangu-Σ model, and is used for Chinese and multilingual language understanding tasks. Official references appear in Huawei Cloud documentation and the Pangu Model GitHub repository.
PanguBot crawls using HTTP/1.1 and supports both HTTP and HTTPS, with a default crawl rate of around 1–2 requests per second per IP, according to observed traffic patterns. Its IP ranges primarily originate from Huawei Cloud’s AS136907 and AS55960, with ranges such as 121.36.0.0/16 and 124.70.0.0/16. It respects a crawl delay of 10 seconds by default when specified in robots.txt, though it can be configured to be more aggressive. The bot follows standard link traversal via and tags, and parses JavaScript minimally; it does not execute JavaScript for content rendering. User-Agent strings include “PanguBot” and variant “PanguBot/1.0”. It may also include the header “X-PanguBot: 1” in requests. Documentation from Huawei Cloud states the crawler avoids password-protected areas and does not log personal data.
PanguBot fully honors robots.txt directives, as confirmed by Huawei Cloud’s official guidance. It will obey both “Disallow” and “Crawl-Delay” tokens. There are no known reports of it ignoring disallow rules. However, it does not support the “Allow” directive for overriding disallowed paths in the same way as Googlebot, so webmasters should place explicit Disallow lines for any restricted content.
The primary User-Agent string is “PanguBot/1.0”, sometimes with platform suffixes like “Windows NT 10.0”. No trailing comment strings are used. The bot sets the “From” header to a Huawei Cloud contact email in rare cases, but not consistently. Behavioral fingerprints include a time-to-live (TTL) of the connection around 30 seconds and a consistent request pattern with a 10-second idle between pages. The bot’s IPs are listed in Huawei Cloud’s public IP range databases (e.g., ip-ranges.amazonaws.com equivalent for Huawei).
Collected data is used exclusively for training language models within the Pangu ecosystem, including Pangu-Σ, which powers Huawei’s AI services such as Pangu Natural Language Understanding. The company states that personal identifiable information (PII) is filtered out before training. The data also supports fine-tuning for enterprise NLP tasks via Huawei Cloud’s ModelArts platform. There is no indication of resale or public redistribution of crawled content.
PanguBot is rate-limited because despite legitimate intent, its sustained crawl can still overwhelm smaller servers if left unmanaged. Policy rationale: threshold-based blocking (e.g., >5 req/sec from a single PanguBot IP) is acceptable to maintain site performance, per Huawei’s own recommendation for webmasters to configure rate limits if needed.
— Imperva Bad Bot Report 2026
How much of your traffic is automated? Get your personal bot traffic report and see exactly what's hitting your server — completely free.
📊 Get My Bot ReportSign up in seconds · No card required
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.