🤖 Overview

scspider is the official web crawler user‑agent string used by Sogou Inc., a major Chinese search engine and subsidiary of Sogou (formerly Sohu). Its primary purpose is to index publicly accessible web pages for Sogou’s search results and other Sogou products, such as Sogou News and Sogou Image Search. According to Sogou’s webmaster guidelines (webmaster.sogou.com), scspider is a legitimate, rate‑limited bot that operates under the same ethical standards as other mainstream search‑engine crawlers.

🌐 Technical Behavior

scspider performs an HTTP/1.1 GET request with the User‑Agent string Sogou web spider/4.0, though variations like Sogou News Spider and Sogou Spider may appear in logs. The crawler respects robots.txt directives and typically fetches a few pages per second from a given server, making it moderately aggressive but manageable with standard rate‑limiting. Its IP ranges are allocated from Sogou’s owned blocks in China (e.g., 61.135.162.0/24, 106.120.0.0/15) and are listed in the IP address sets published by Sogou’s operations team on their official website. The crawler supports the Accept‑Encoding header (gzip, deflate) and uses a request interval that can increase if the server responds with 503 or 429 status codes.

📋 robots.txt Compliance

scspider is documented to honor robots.txt Disallow directives. Sogou’s webmaster portal explicitly states that the crawler checks the robots.txt file before each crawl and will not access paths you block. However, there have been community reports (e.g., in Chinese technical forums) that scspider sometimes fails to respect all rules when crawling large sites, but these are edge cases and Sogou has addressed them through updates.

🔍 Detection Indicators

The primary detection indicator is the User‑Agent string Sogou web spider/4.0 (or Sogou News Spider). Additional fingerprints include the lack of a Referer header in most requests and the use of a small set of IP addresses from the ASN 4837 (China Unicom) and ASN 23650 (Sogou). The bot also sends a From header containing [email protected] — an email address for abuse reports — which is a strong behavioral fingerprint.

📊 Data Usage

Collected data is used exclusively for Sogou’s search engine index, news aggregation, and image search. Sogou does not publicly disclose use of crawler data for AI training, but given the company’s focus on NLP (their “Sogou Input Method” and “Sogou AI”), it is plausible that indexed data supports machine‑learning models. Sogou’s privacy policy (sogou.com/privacy) outlines that they treat crawl data as public index content.

⚙️ Rate Limiting Policy

Because scspider can send dozens of requests per minute to popular sites, it is rate‑limited to prevent excessive load. A threshold of 10 requests per second per IP is typically sufficient. The policy rationale is that without rate limiting, the crawler could degrade performance for other visitors, especially on shared hosting.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.