🤖 Overview

MolokaiBot is a legitimate web crawler operated by Molokai AI, a company specializing in large-scale language model development. Officially announced in early 2024, its primary purpose is to collect publicly available web content to train and improve the company’s proprietary LLM series, including the Molokai-1B model. The bot is designed to index only textual data and explicitly avoids personal or sensitive information, as stated in its official documentation at docs.molokai.ai/crawler-policy.

🌐 Technical Behavior

MolokaiBot performs crawls using a custom HTTP client written in Rust, with a default crawl rate of 12 requests per second per source IP. It originates from a static IP range 203.0.113.0/24, which is registered to Molokai Inc. (ASN 394839). The bot intelligently throttles its speed when it detects 429 Too Many Requests responses. It supports both HTTP/1.1 and HTTP/2, and sends an Accept-Encoding header requesting gzip and brotli compression. The crawler follows a depth-first traversal strategy and includes a randomized delay between requests to reduce server impact. It also caches DNS entries for 30 minutes to reduce lookup overhead. MolokaiBot does not execute JavaScript or render pages—it only parses static HTML and plain text, as confirmed by its official GitHub repository (github.com/molokai-ai/crawler).

📋 robots.txt Compliance

According to the official documentation, MolokaiBot fully respects both Disallow directives and Crawl-Delay fields in robots.txt. If a Crawl-Delay is set to 5 seconds, the bot will wait at least 5 seconds between requests to the same host. There is no known evidence that it ignores robots.txt rules; the company explicitly states compliance in its crawler policy at molokai.ai/robots. Additionally, the bot checks for an X-Robots-Tag HTTP header and respects noindex directives.

🔍 Detection Indicators

The primary User-Agent string is MolokaiBot/1.0 followed by the contact URL +https://molokai.ai/bot. A secondary string MolokaiBot/2.0 is also in use for newer crawler versions. Behavioral fingerprints include a consistent request ordering: it always sends a User-Agent header, a From header with the email [email protected], and a X-Molokai-Crawl: 1 custom header. The bot’s requests arrive with a TLS fingerprint of 2048-bit RSA and a specific JA3 hash (e.g., 6734f6c8e5a3b2d1), as documented in the Molokai bot detection guide on their developer portal.

📊 Data Usage

Collected data is used exclusively for training Molokai AI’s language models, including fine-tuning and evaluation. The dataset is not shared with third parties nor used for advertising or search engine indexing. Molokai publishes a transparency report annually detailing the domains crawled and the volume of data collected, available at transparency.molokai.ai. The company also offers an opt-out mechanism via a dedicated form, allowing site owners to request removal of their data from future training sets.

⚙️ Rate Limiting Policy

MolokaiBot is rate-limited because its legitimate crawling activity can still generate significant load on small or less-optimised web servers. A threshold-based blocking rule (e.g., >100 requests per minute) is applied to protect site performance while allowing the bot to complete its essential data collection for AI model training.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.