🤖 Overview

The fetch_ici crawler is operated by the International Computing Institute (ICI), a non-profit research organization based in Switzerland. It was first publicly documented in 2022 as a dedicated web crawler for building large-scale, multi-lingual text corpora used to train open‑source natural language processing models. Its primary product is the ICI-Corpus, a dataset of cleaned web pages distributed under a Creative Commons license. According to ICI’s official documentation (https://ici.org/crawler), the bot collects only publicly accessible text and discards non‑textual content such as images and scripts to comply with fair‑use guidelines.

🌐 Technical Behavior

The crawler uses a headless Chromium-based browser (via Puppeteer) to render JavaScript-heavy pages, and it respects HTTP/2 and HTTP/3 protocols. Its request frequency is configurable but typically defaults to 5 requests per second per IP, with a 2‑second random jitter between bursts. IP ranges are announced via ICI’s ASN (AS208196) and are publicly listed at https://ici.org/ip-ranges. The crawler issues GET requests with Accept‑Encoding: gzip, deflate, br and uses a rotating user‑agent pattern to avoid blocking, but the base string remains constant. It also sends a custom X‑ICI‑Crawl‑ID header with a UUID for traceability. The bot does not follow nofollow links but does obey rel="nofollow" on individual anchors when specifically configured.

📋 robots.txt Compliance

ICI’s documentation explicitly states that fetch_ici honors robots.txt Disallow directives as specified in the Robots Exclusion Standard (RFC 9309). The crawler reads the file at startup and recaches it every 24 hours. ICI runs a public validator at https://ici.org/robots‑check that shows which paths are blocked. There is no documented evidence of deliberate non‑compliance; however, independent researchers have noted that the bot may ignore Crawl‑Delay directives if they are set below 1 second, treating them as advisory rather than mandatory.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; fetch_ici/1.0; +https://ici.org/bot). A secondary string, fetch_ici‑research/1.0, is used exclusively for academic mirrors. Behavioral fingerprints include the X‑ICI‑Crawl‑ID header and a consistent pattern of requesting /robots.txt before any other resource. The bot also sends a From header containing [email protected]. Network administrators can verify the bot by reverse‑DNS lookup: all IPs resolve to *.crawl.ici.org.

📊 Data Usage

Collected data is entirely used for AI training and academic research. Specifically, the ICI‑Corpus fuels multilingual transformer models (such as the ICI‑BERT series) and is also shared with partner universities under data‑sharing agreements. The corpus is cleaned of personally identifiable information (PII) via a pipeline that removes email addresses, phone numbers, and credit‑card patterns. ICI publishes annual transparency reports detailing the number of domains crawled and the volume of data ingested.

⚙️ Rate Limiting Policy

Although fetch_ici is legitimate and well‑behaved, it is still rate‑limited by many webmasters because its default crawl rate of 5 requests per second can overwhelm smaller servers. The policy rationale for threshold‑based blocking is to protect server resources while still allowing the bot access — for example, reducing the permitted request rate to 1 request per second ensures the site remains responsive for human visitors without completely denying the crawler’s data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.