🤖 Overview

ivia is a legitimate web crawler operated by Ivia Technologies Inc., a data services company focused on collecting publicly available web content for training proprietary AI language models and enhancing natural language processing research. First documented in early 2022, the crawler's primary purpose is to feed a curated dataset used to improve Ivia's IviaLM model and analytics platforms. The bot is explicitly listed in Ivia's official documentation at docs.ivia.ai as a non-malicious agent designed for scalable web indexing.

🌐 Technical Behavior

The crawler employs asynchronous HTTP requests via a custom library built on aiohttp, typically executing between 10 to 50 requests per second per IP with a default crawl depth of 3 links. It respects Cache-Control headers and uses a rotating pool of IPv4 and IPv6 addresses primarily sourced from AWS (EC2 and Lambda) and Google Cloud Platform. Traffic is distributed across data centers in North America, Europe, and Asia, supporting both HTTP/1.1 and HTTP/2 protocols. The bot identifies itself via the User-Agent string ivia/1.0 (sometimes ivia/1.1) and does not execute JavaScript, parse dynamic content, or follow meta refresh redirects. It uses a custom header X-IA-Crawler: true in verified implementations.

📋 robots.txt Compliance

According to Ivia's published guidelines on their GitHub repository (github.com/ivia/crawler), the ivia crawler fully honors Disallow directives in robots.txt and respects Crawl-Delay instructions when present. However, it may ignore Allow rules if a Disallow is explicitly set for the same path, a behavior documented in their official stance. The crawler also checks for X-Robots-Tag HTTP headers as an additional compliance measure.

🔍 Detection Indicators

Primary detection relies on the User-Agent string ivia/1.0 or ivia/1.1, paired with the absence of typical browser headers like Accept-Language and Sec-Fetch-Site. The custom header X-IA-Crawler: true is a strong behavioral fingerprint. IP ranges are published by Ivia at ivia.ai/ips.txt and belong to the ASNs of AWS (146.18.0.0/16) and GCP (35.192.0.0/12). No CVE entries are associated with this bot as it has no known vulnerabilities.

📊 Data Usage

Collected data is used exclusively for training Ivia's proprietary large language model IviaLM and for improving the company's search and analytics products. Ivia explicitly states in their privacy policy that personally identifiable information (PII) is filtered during preprocessing and not stored in training datasets. Data is also shared with academic research partners under non-disclosure agreements for NLP advancements.

⚙️ Rate Limiting Policy

Although the ivia crawler is a legitimate agent, it is subject to rate limiting in production environments because its high request volume (up to 50 req/s per IP) can degrade server performance if left unchecked. Threshold-based blocking, typically set at 100 requests per minute per IP, ensures fair resource allocation while allowing Ivia to collect necessary data for model training.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.