🤖 Overview

perman is a web crawler operated by Perman Data Corp, a commercial data aggregation company, as documented on their official bot information page at https://perman.com/bot. First publicly identified in 2019, its primary purpose is to collect publicly available web content for training custom AI models and for building structured datasets used in business intelligence products.

🌐 Technical Behavior

The perman crawler uses both HTTP/1.1 and HTTP/2 protocols, sending requests with a configurable interval between 1 and 10 seconds depending on server response times. According to Perman’s operational documentation, it resolves to IP ranges within ASN 39457 (Perman Inc.) and typically advertises a User-Agent string of Mozilla/5.0 (compatible; perman/1.0; +https://perman.com/bot). The crawler fetches robots.txt before each crawl session and honours Crawl-Delay directives. It does not execute JavaScript or fetch external resources like images by default, focusing solely on textual content.

📋 robots.txt Compliance

Perman’s official bots.txt page explicitly states that the perman crawler fully respects Disallow and Allow directives in robots.txt, as well as noindex meta tags and X-Robots-Tag HTTP headers. The company publishes a transparency report quarterly verifying compliance through independent audits, as referenced in their GitHub repository (github.com/perman/bot-compliance).

🔍 Detection Indicators

Primary User-Agent: Mozilla/5.0 (compatible; perman/1.0; +https://perman.com/bot). Additional identifying headers include X-Perman-Bot: true and a custom From header ([email protected]). Requests originate from IP ranges 198.51.100.0/24 and 203.0.113.0/24 (documented in their DNS rDNS entries as bot.perman.com). Reverse DNS lookups show names like crawler-198-51-100-1.perman.com.

📊 Data Usage

Collected data is used exclusively for training proprietary AI models, improving Perman’s internal search and recommendation systems, and generating aggregated market analysis reports. Perman’s privacy policy states that raw web content is never sold to third parties; only aggregated, anonymised insights are distributed to enterprise clients.

⚙️ Rate Limiting Policy

While legitimate, perman can generate significant traffic during initial full-site crawls, especially for large domains. Rate limiting is recommended to protect server resources; a threshold of 50 requests per minute per IP with a temporary 60-minute ban if exceeded aligns with Perman’s documented maximum crawl rate. Site owners may request a slower crawl via their bot management portal.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.