🤖 Overview

Majestic-SEO is a web crawler operated by Majestic, a UK-based digital marketing company founded in 2004 and acquired by M-Files in 2018. Its primary purpose is to build the Majestic Site Explorer and Fresh Index, a backlink database used for SEO analysis, competitor research, and link intelligence. The bot feeds data into Majestic’s proprietary index, which tracks over 5 trillion historical URLs and provides metrics like Citation Flow and Trust Flow.

🌐 Technical Behavior

Majestic-SEO crawls websites using an asynchronous, distributed architecture with multiple IP addresses from a pool that includes 64.71.137.0/24, 208.87.240.0/24, and other ranges owned by Pegasus Tech Ventures. It requests pages at a variable rate, typically between 1–5 requests per second per IP, but can burst up to 20 requests per second during index refreshes. The bot follows HTTP/1.1 keep-alive connections and respects robots.txt directives but does not support Crawl-delay directives natively—instead it uses its own rate-limiting algorithm based on site responsiveness. The crawler primarily targets HTML pages and .pdf/.doc files for link extraction, but does not fetch JavaScript or CSS content. According to Majestic’s official documentation, the bot uses a custom Majestic-SEO User-Agent with a version number (e.g., Majestic-SEO/2.0) and also sends the X-Majestic-Visit header containing a unique visit ID.

📋 robots.txt Compliance

Majestic-SEO fully honors Disallow directives in robots.txt as documented in their official crawling policy page (https://majestic.com/help/robots). Majestic states that they “will not crawl any URL disallowed in robots.txt” and also respect Allow overrides. However, they do not support the Crawl-delay directive; instead, they apply a dynamic delay based on server response times. For rate limiting, site owners can use Majestic-SEO’s own Access Control settings via the Majestic dashboard to add custom exclusions.

🔍 Detection Indicators

The primary User-Agent string is Majestic-SEO/2.0 (compatible; Majestic-12; http://majestic12.co.uk/bot.php). Additional strings include MJ12bot/v2.0 (legacy) and Majestic-SEO/1.0. The bot also sends a From header with the value [email protected]. Behavioral fingerprints include requests for robots.txt at the root, high frequency of HEAD requests before GET requests, and a tendency to crawl URLs with query parameters in depth-first order. The bot does not send a Referer header.

📊 Data Usage

Collected data—specifically backlink profiles, anchor text, and page metadata—is used to populate the Majestic Site Explorer and Fresh Index. This data powers SEO tools for link analysis, authority scoring, and domain comparison. Majestic also provides an API for third-party integrations. No content is used for AI training; the focus is strictly on link graph construction.

⚙️ Rate Limiting Policy

Because Majestic-SEO crawls aggressively during large index updates (e.g., Fresh Index refreshes every 24 hours), it is **rate-limited** to prevent server overload. Threshold-based blocking (e.g., > 20 req/s from a single IP) is applied to protect site availability while still allowing the bot to collect necessary link data.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.