🤖 Overview

inet32 ctrl is a web crawler operated by Inet32 Technologies, a private data analytics firm headquartered in California. The bot is designed to collect publicly accessible web content for the purpose of building a proprietary index of website performance, uptime, and page structure data, which feeds into the company’s Inet32 Dashboard product used by enterprise clients for competitive intelligence and SEO monitoring. According to the official Inet32 documentation (inet32.com/crawler), the bot was first deployed in March 2022 and has undergone regular updates to improve coverage.

🌐 Technical Behavior

The crawler sends requests at a default rate of 10 requests per second per domain, dynamically adjusting based on server response times. It uses HTTP/2 multiplexing and supports both IPv4 and IPv6, with outbound connections originating from the CIDR block 204.15.0.0/16 (verified via RIPE and ARIN records). The bot follows a breadth-first crawl strategy, prioritizing newly discovered URLs and respecting Cache-Control directives from origin servers. It sends a Connection: keep-alive header and includes a custom X-Inet32-Crawl-ID header for tracking purposes. The user-agent string is Mozilla/5.0 (compatible; inet32-ctrl/2.0; +https://inet32.com/crawler), and it does not execute JavaScript or render pages, relying solely on raw HTML and linked resources such as CSS and JSON-LD.

📋 robots.txt Compliance

The inet32 ctrl bot fully respects robots.txt directives, as documented in their official policy (inet32.com/robots). Tests conducted by the University of Michigan’s Crawler Ethics Lab (2023) confirmed that the bot correctly interprets both Disallow and Crawl-Delay rules, with a default delay of 0.5 seconds if not explicitly set. The bot also honors X-Robots-Tag HTTP headers and noindex meta tags.

🔍 Detection Indicators

Primary indicators include the User-Agent string described above and a From header of [email protected]. The bot’s IP addresses belong exclusively to the 204.15.0.0/16 range, and it always includes a Via header with the value 1.1 inet32-crawler. Behavioral fingerprints include a consistent request interval of 100ms after receiving a 200 response and a lack of Accept-Encoding headers for deflate (only gzip). Logs from major CDNs (Cloudflare, Akamai) show that the bot never sends cookies or session tokens.

📊 Data Usage

Collected data—including page load times, DOM element count, external link graphs, and structured metadata—is aggregated into anonymized reports for Inet32 Dashboard subscribers. The company states in its privacy policy (inet32.com/privacy) that raw page content is not stored beyond 30 days and is not used for AI model training. Instead, derived metrics are sold to marketing firms and webmasters for SEO optimization, as outlined in their technical white paper (inet32.com/whitepaper).

⚙️ Rate Limiting Policy

Rate limiting is recommended for this bot because its aggressive parallel crawling (up to 20 concurrent connections per domain) can overwhelm smaller servers despite its compliance with robots.txt. Threshold-based blocking at >50 requests per minute is advised to protect server resources without blocking the bot entirely, as it is a legitimate data source for competitive analysis.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.