🤖 Overview

MojeekBot is the proprietary web crawler operated by Mojeek, a UK-based independent search engine headquartered in Brighton, England, founded in 2004 by Marc Smith. Its primary purpose is to build and maintain Mojeek’s own search index, which is distinct from major search engines by not relying on Google or Bing data, thereby offering users a privacy-focused alternative with no tracking, no profiling, and no ads. The bot systematically discovers and downloads publicly available web pages to feed into Mojeek’s proprietary search algorithm, which emphasizes unbiased results and user anonymity.

🌐 Technical Behavior

MojeekBot employs a polite, crawl-delay-respecting strategy, typically issuing requests at a frequency that aligns with the site’s Crawl-Delay directive in robots.txt or a default rate of approximately one request per 10–15 seconds. The bot supports both HTTP/1.1 and HTTP/2 protocols, and it identifies itself via the User-Agent string "MojeekBot/0.1 (archi...)" as documented on Mojeek’s official website. Its IP ranges are varied and include addresses from several UK-based hosting providers, such as those in the 185.17.184.0/23 subnet, and it frequently rotates through a small pool of addresses to distribute load. The crawler primarily follows links from already indexed pages and submits new or updated URLs via Mojeek’s submission API, also respecting HTTP Last-Modified headers and Etag values to avoid re-crawling unchanged content.

📋 robots.txt Compliance

Mojeek officially states that MojeekBot fully respects robots.txt directives, including both Disallow and Allow rules. Verified via Mojeek’s own documentation at https://www.mojeek.com/about/search-engine/indexing/bot.html, the bot reads the robots.txt file before each crawl session and honors custom crawl-delay settings. There are no known documented violations or controversies regarding its compliance.

🔍 Detection Indicators

The primary detection method is the User-Agent string: "MojeekBot/0.1 (archi...)" often appended with version information such as "MojeekBot/0.1 (archi...; http://www.mojeek.com/bot.html)". Behavioral indicators include a consistent request pace reflecting the site’s Crawl-Delay, no JavaScript execution, and the absence of session cookies. Additionally, the bot typically sends a From header with the Mojeek contact email and a Referer header set to the Mojeek homepage for its first request on a domain.

📊 Data Usage

All data collected by MojeekBot is exclusively used to populate the Mojeek search index, which powers the search results displayed on mojeek.com. The company explicitly states that no collected content is sold, shared with third parties, or used for AI/ML training, advertising profiles, or any purpose other than indexing and ranking for its own search engine. This commitment is part of Mojeek’s privacy policy publicly available at https://www.mojeek.com/about/privacy/.

⚙️ Rate Limiting Policy

Rate limiting is applied to MojeekBot only when it exceeds the specified Crawl-Delay or when a site experiences unusually high load due to misconfiguration. The rationale for threshold-based blocking is to prevent any single crawler from overwhelming web servers while still allowing comprehensive indexing; Mojeek encourages sites to set a reasonable crawl delay in robots.txt rather than blocking the bot entirely.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.