🤖 Overview

HaosouSpider is a web crawler operated by Qihoo 360 Technology Co. Ltd., the Chinese cybersecurity and internet company behind the Haosou search engine (formerly 360 Search, accessible at haosou.com). Officially documented on Qihoo 360’s crawler support page (http://www.haosou.com/help/help_3_2.html), its primary purpose is to index publicly accessible web pages for Haosou’s search results, delivering relevant links to users across China. The bot is a core component of Haosou’s search infrastructure, which competes with Baidu and Sogou in the Chinese market, and collects data exclusively for search indexing—not for AI training or analytics.

🌐 Technical Behavior

HaosouSpider employs a breadth‑first crawl strategy, requesting pages in rapid succession from a distributed fleet of IP addresses allocated to Qihoo 360. Official sources list its IP ranges as belonging to AS4808 (China Unicom) and AS4837 (ChinaNet), with specific blocks such as 123.125.71.* and 106.120.168.* frequently observed. The bot sends HTTP/1.1 GET requests with a default crawl interval of approximately 1–3 seconds per host, though this can slow under robots.txt restrictions. It supports both HTTP and HTTPS protocols, and its requests include standard headers like Accept: text/html,application/xhtml+xml and a Connection: keep-alive header. HaosouSpider does not fetch embedded resources (e.g., images, CSS) unless explicitly linked, and it follows redirects (up to five hops) before discarding a URL.

📋 robots.txt Compliance

Based on Qihoo 360’s published guidelines, HaosouSpider honors the robots.txt file if it exists and correctly parses Disallow directives for the user-agent HaosouSpider or 360Spider. The official documentation (http://www.haosou.com/help/help_3_2.html) states that site owners can block the crawler using User-agent: HaosouSpider followed by Disallow: /. Third‑party tests confirm that over 95% of servers respecting robots.txt see the crawler cease access within 30 minutes of a change. However, the bot has been known to ignore malformed or empty robots.txt files, and it does not support the Crawl-delay directive, instead implementing its own rate‑limiting based on server response times.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; HaosouSpider/1.0; +http://www.haosou.com/help/help_3_2.html). Occasionally, variations like 360Spider or Sosospider (a legacy alias) appear. Behavioral fingerprints include a high request frequency from Chinese IP ranges, a lack of JavaScript execution, and a typical referer header of http://www.haosou.com/. The bot rarely sends a From or X-Forwarded-For header, and its HTTP user‑agent field is always followed by an explanation URL pointing to the Haosou help page.

📊 Data Usage

All data harvested by HaosouSpider is used exclusively for building and updating Haosou’s search index. The crawled content—HTML, metadata, and link structures—feeds the ranking algorithm that determines page prominence in search results. No content is repurposed for AI model training, advertising profiling, or third‑party distribution, as per Qihoo 360’s privacy policy (https://www.360.cn/privacy/). The index refresh cycle is reported to be 24–48 hours for high‑authority domains, while smaller sites may see updates weekly.

⚙️ Rate Limiting Policy

HaosouSpider is rate‑limited because its distributed crawling can overwhelm under‑provisioned web servers, particularly those with shared hosting or limited bandwidth. Many operators enforce a threshold of 100 requests per minute per IP block to prevent resource exhaustion, a policy aligned with the bot’s own compliance with robots.txt but not with explicit Crawl-delay directives.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.