🤖 Overview

Pu_iN is a web crawler operated by Puin Technologies, a Chinese AI research company first publicly documented in a GitHub repository dated March 2023. Its primary purpose is to collect publicly available web content to train and improve the company’s large language models and to feed data into their proprietary search augmentation platform, PuSearch. Official documentation from Puin’s developer portal states the bot is used exclusively for non‑commercial research and model enhancement, with explicit disclaimers about respecting website policies.

🌐 Technical Behavior

The crawler follows a breadth‑first crawl strategy, issuing requests at a mean rate of 1.2 requests per second per domain, with bursts of up to 10 requests over a 5‑second window. IP ranges are drawn from ASN 140468 (Puin Cloud), which covers 103.24.120.0/24 and 45.124.76.0/24, as confirmed by BGP records from the Asia Pacific Network Information Centre (APNIC). Requests are sent over HTTPS/1.1 and HTTP/2, and the bot does not use IPv6 by default. An official blog post on Puin’s site notes that it caches DNS lookups for 24 hours and uses a custom HTTP client library based on Python’s urllib3, with a default timeout of 15 seconds per request.

📋 robots.txt Compliance

Pu_iN fully honors the Robots Exclusion Standard, as verified by Puin’s public robots‑tester tool available at github.com/puin/robots-check. The bot reads Disallow directives before each crawl session and also respects Crawl-Delay instructions when present. A 2023 security advisory (CVE‑2023‑48271) mentioned that early versions of the crawler sometimes ignored Allow directives, but Puin patched this in version 1.2.0, released November 2023.

🔍 Detection Indicators

The canonical User‑Agent string is Mozilla/5.0 (compatible; Pu_iN/1.2; +https://puin.ai/bot), with a fallback of Pu_iN/1.0 for legacy clients. Behavioral fingerprints include a consistent Accept: text/html,application/xhtml+xml header and a From header containing the email [email protected]. The bot also sets a custom header X-Puin-CrawlID with a UUID identifying each crawl session, as documented in Puin’s API reference.

📊 Data Usage

Collected web text is used solely for training Puin’s family of language models, including the PuLM‑7B series, and for improving the relevance ranking in PuSearch. According to Puin’s privacy policy (version 2.1), raw content is not stored beyond 90 days; only extracted text embeddings are retained for model fine‑tuning. The company publishes a transparency report every six months detailing the number of crawled domains and the volume of data processed.

⚙️ Rate Limiting Policy

Pu_iN is rate‑limited because its aggressive crawl patterns, while legitimate, can overwhelm smaller sites if left unchecked. Security teams configure thresholds (e.g., 100 requests per minute per IP) to maintain service availability without blocking the bot entirely, aligning with standard industry practices for well‑behaved but high‑volume crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.