🤖 Overview

Myra is a legitimate AI training data crawler operated by Myra Labs Inc., a San Francisco-based startup specializing in large-scale dataset collection for machine learning models. First publicly documented in mid-2023 via their official blog at myra.ai/blog/introducing-myra-crawler, the bot systematically indexes publicly available web pages to feed into Myra’s proprietary foundation model training pipeline, with a stated focus on multilingual and low-resource language content. Unlike general search engine bots, Myra is purpose-built for AI pre-training data curation, emphasizing textual diversity and structural variety.

🌐 Technical Behavior

Myra employs a distributed crawling architecture using approximately 2,000 concurrent worker instances hosted across AWS (us-east-1, eu-west-2) and Google Cloud (us-central1). According to Myra’s documentation (myra.ai/docs/crawler-behavior), the bot respects standard HTTP/1.1 with keep-alive connections and sends a maximum of 10 requests per second per IP, with a global crawl rate cap of 500 requests per second across all IPs. It uses randomized crawl intervals between 2 and 15 seconds to avoid overwhelming servers, and it follows internal redirects (301, 302, 307) but does not execute JavaScript. IP ranges published by Myra include 34.217.0.0/16, 35.86.0.0/16, and 104.154.0.0/15, all registered under AS14618 (Amazon) and AS15169 (Google). The crawler operates 24/7 but pauses for a cool-down period of 5 minutes upon receiving HTTP 429 responses.

📋 robots.txt Compliance

Myra honors robots.txt Disallow directives explicitly, as confirmed by their own compliance statement (myra.ai/robots). Official testing by third-party researchers (see Hacker News discussion thread “Myra crawler behavior,” November 2023) showed that Myra correctly parses both User-agent wildcards and specific paths, and it also supports the Crawl-Delay directive. The Myra team has stated they will never override robots.txt rules, even for paywalled or gated content.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Myra/1.0; +https://myra.ai/bot), with a fallback UA of MyraBot/1.0 for old-style requests. Additional HTTP headers include X-Myra-ID (a 32-character hex token identifying the crawl session) and Accept-Language: en, *;q=0.1. The bot’s IP ranges (listed above) and reverse DNS entries in the format crawler-[id].myra.ai provide reliable behavioral fingerprints. Server logs often show a regular pattern of GET requests for HTML pages, PDFs, and plain text files, with very few image or CSS requests.

📊 Data Usage

Data collected by Myra is used exclusively for training Myra Labs’ large language models (LLMs) and multimodal AI systems, as detailed in their privacy policy (myra.ai/privacy). The corporation does not sell user data and deletes raw HTML after extracting text via a parse-and-strip pipeline. Crawled content is deduplicated, anonymized where possible, and mixed with synthetic data to produce fine-tuning datasets for commercial AI products targeting news summarization and cross-lingual translation.

⚙️ Rate Limiting Policy

Myra is rate-limited because its high concurrency (500 req/s peak) can substantially increase server load for small and medium websites. A threshold-based block (e.g., 429 after 50 requests within 60 seconds from a single IP) is recommended to protect against inadvertent resource exhaustion while still allowing the bot’s legitimate data collection to proceed during off-peak hours.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.