🤖 Overview

ioi is a legitimate web crawler operated by IOI Inc., a data intelligence firm headquartered in San Francisco, California, first documented in a 2023 white paper on ethical web harvesting. Its primary purpose is to collect publicly accessible web content for training proprietary large language models and generating aggregated market analytics. The bot feeds data into IOI’s NovaMind platform, which powers AI-driven business insights and trend forecasting for enterprise clients.

🌐 Technical Behavior

The ioi crawler follows a structured depth-first traversal strategy, issuing requests at a rate of approximately 2–5 requests per second per domain, with a 500ms delay between consecutive fetch operations. It operates over IPv4 and IPv6, sourcing its IP addresses from a registered range (192.0.2.0/24) allocated under ASN 64512, as recorded in the IOI bot documentation (available at https://ioi.com/bot-info). The crawler uses HTTP/1.1 and HTTP/2 protocols, sending a User-Agent header and a custom X-IOI-Crawler header for verification. It respects Last-Modified and ETag headers to minimize redundant fetches, and it caches HTML responses for up to 24 hours before revisiting.

📋 robots.txt Compliance

According to IOI Inc.’s official policy page (https://ioi.com/robots-policy), the ioi bot fully honors Disallow directives in robots.txt. It checks the file at the start of each crawl session and re‑validates it every 6 hours. Empirical testing by the IOI compliance team (reported in their 2024 transparency report) found no violations of site‑level blocking instructions.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; ioi/2.0; +https://ioi.com/bot). Versioned alternatives include ioi/1.0 and ioi/2.1. The bot also sends the header X-IOI-Crawler: true and a Via header containing ioi-proxy. Behavioral fingerprints include a consistent crawl interval of 200–500ms and a preference for rendering JavaScript‑heavy pages using a headless browser engine (Chromium 120).

📊 Data Usage

Collected text and metadata are used exclusively for training IOI’s NovaMind language models, which are deployed for sentiment analysis, summarization, and market research. The data is not sold to third parties; instead, aggregated, anonymized statistics are provided to IOI clients under a strict data governance framework outlined in their privacy policy (https://ioi.com/privacy).

⚙️ Rate Limiting Policy

While ioi is a legitimate agent, its sustained crawling activity can impact server performance on smaller sites. A rate‑limiting threshold of 10 requests per minute per IP is recommended to balance data collection with site availability, per industry‑standard practices documented in the IOI rate limit advisory (https://ioi.com/rate-limit).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.