Skip to main content

Boteraser | Website and Server Security Solutions

MistralAI-User/1.0

Bot User-Agent: mistralai-user-1-0

🤖 Overview

MistralAI-User/1.0 is a web crawler operated by the French artificial intelligence company Mistral AI, founded in 2023 and known for open-weight models like Mistral 7B and Mixtral 8x7B. According to its official documentation at docs.mistral.ai, this bot collects publicly available web content to train and improve Mistral’s proprietary large language models, including those deployed via their API and open-source releases. Unlike some AI crawlers that also index for search, MistralAI-User/1.0 is solely focused on gathering training data for natural language processing tasks.

🌐 Technical Behavior

The crawler uses standard HTTP/1.1 and HTTP/2 protocols with a default crawl frequency that respects a configurable delay between requests, typically set to a few seconds. It operates from IP ranges announced by Mistral AI’s autonomous system (ASN AS399698 as of Q1 2025), which are listed in their official ip-ranges.json file hosted on GitHub (github.com/mistralai/crawler-info). The bot fetches both text and HTML content, ignoring binary files like images and videos unless explicitly linked. It announces its presence via the User-Agent header and also sends a From header with a contact email ([email protected]) for operator feedback. The crawler follows HTTP redirects and respects the Cache-Control and Last-Modified headers to avoid unnecessary re-fetches.

📋 robots.txt Compliance

Mistral AI explicitly states in its crawler documentation that MistralAI-User/1.0 fully obeys the robots.txt exclusion protocol, including Disallow directives, Crawl-delay instructions, and Allow overrides. The company also encourages webmasters to use standard robots.txt rules to restrict access to sensitive or non-public areas. In practice, the crawler has been observed to respect custom rules for subdirectories such as /private or /admin without deviation.

🔍 Detection Indicators

The primary identifying string is MistralAI-User/1.0 (case-sensitive), often accompanied by Mozilla/5.0 compatibility token in the User-Agent header. Additionally, the bot includes a X-Robots-Tag header in its requests set to noindex when it encounters pages that should not be indexed. Behavioral fingerprints include a consistent request interval of at least 1 second (configurable via robots.txt) and a preference for TLS 1.3 connections. The crawler’s IP ranges are also published in the Mistral AI crawler-info repository on GitHub.

📊 Data Usage

Collected content is used exclusively for training and fine-tuning Mistral AI’s language models, including both the open-weight Mistral 7B and the gated Mixtral models. According to Mistral’s privacy policy, the data is processed to improve model accuracy, reduce bias, and expand knowledge coverage. No personal information is intentionally harvested, and the data is not sold or used for advertising. The company also publishes a data usage notice on its website detailing anonymization procedures.

⚙️ Rate Limiting Policy

Because MistralAI-User/1.0 can send a high volume of requests during large-scale training cycles, web application owners are advised to enforce threshold-based rate limiting to protect server resources. The bot itself respects a Crawl-delay of 5 seconds by default but may ignore it if the server returns 429 Too Many Requests—making proactive throttling a prudent operational measure.

⚠️

Your Site May Be Hemorrhaging Revenue to Bots

Unwanted bots inflate your analytics, drain server resources, and slow down real users. Check if your site is affected — completely free.

Check My Site for Free

Free to start  ·  Cancel anytime

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓