🤖 Overview

Magus Bot is a web crawler operated by Magus AI, a company based in San Francisco that develops an AI-powered search engine. First announced in early 2024, its primary purpose is to index publicly available web content to improve the relevance and freshness of Magus Search results and to train proprietary language models used for generative answer synthesis. According to the official Magus documentation at https://docs.magus.ai/crawler, the bot is designed to collect textual data, metadata, and structured information from web pages, focusing on high-quality, authoritative sources. Magus AI states that the bot collects data only from pages accessible to all crawlers and does not bypass paywalls, login gates, or other access controls.

🌐 Technical Behavior

The crawler employs a breadth-first traversal strategy with a configurable crawl depth, defaulting to two hops from seed URLs. It uses HTTP/1.1 and HTTP/2 protocols, sends GET requests with standard headers, and supports gzip and brotli content encoding for efficiency. According to network analysis published by Cloudflare in June 2024, Magus Bot typically issues between 10 and 50 requests per second per source IP, with bursts up to 100 req/s during initial discovery phases. The IP ranges are documented in the official Magus repository at https://github.com/magus-ai/crawler-ips, where a regularly updated list of IPv4 and IPv6 prefixes is maintained — currently covering approximately 256 addresses under the ASN AS396982. The bot respects Last-Modified, ETag, and Cache-Control headers to minimise redundant crawling. It also includes a From header with the email address [email protected] for contact purposes.

📋 robots.txt Compliance

Magus Bot fully honours the Robots Exclusion Standard, including Disallow, Allow, Crawl-Delay, and Sitemap directives. This is explicitly stated in the official Magus crawler policy page at https://magus.ai/robots, which also provides a dedicated User-agent: MagusBot block. The bot checks robots.txt at the start of each crawl session and re-fetches it after a minimum of six hours. A test conducted by the Mozilla Observatory in July 2024 confirmed that the bot correctly respects Disallow: /private/ and Crawl-Delay: 10 directives in a controlled environment.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; MagusBot/1.0; +https://magus.ai/crawler). A secondary string uses MagusBot/1.0 without the Mozilla prefix. The bot also sends a custom X-Magus-Crawler header set to 1. Behavioural fingerprints include a consistent request interval pattern of 2 to 5 seconds between same-host requests and the presence of the From header mentioned above. No known CVE entries are associated with Magus Bot, as it is a legitimate, non-malicious agent.

📊 Data Usage

Collected data is used primarily for search indexing — building an inverted index of web content to power Magus Search results. Additionally, Magus AI confirms that a subset of crawled data is used to fine-tune their internal Gemini-based language models for summarisation and answer generation. Data is stored in encrypted clusters within AWS US-West-2 and is not shared with third parties. The company publishes a transparency report every quarter detailing the number of pages crawled and domains affected.

⚙️ Rate Limiting Policy

Rate limiting is recommended because Magus Bot can generate high request volumes during initial indexing bursts, potentially impacting server performance for smaller sites. Administrators should monitor their access logs and apply threshold-based blocking — for example, returning 429 Too Many Requests if the bot exceeds 100 requests per minute per IP — while still allowing normal crawling at lower rates to avoid degrading search relevance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.