🤖 Overview

moget is the web crawler operated by Mojeek, a UK-based independent search engine company founded in 2004 by Marc Smith. Its primary purpose is to index web pages for Mojeek’s search engine, which is one of the few with its own crawler and index, avoiding reliance on third-party search APIs. Moget collects publicly accessible web content to build and maintain Mojeek’s search index, which is used to provide unbiased, privacy-respecting search results.

🌐 Technical Behavior

Moget crawls websites using standard HTTP/1.1 and HTTPS protocols, following links recursively from seed URLs. According to Mojeek’s official documentation at mojeek.com/about/crawler/, the crawler respects standard crawl delays and typically issues requests at a rate of about one request per second per host, but this can vary based on server response times. The crawler’s IP ranges are not publicly documented in a single list, but Mojeek has stated that its crawler IPs are dynamic and that they prefer operators to whitelist by user-agent rather than IP. Moget identifies itself via the User-Agent string "Moget/4.0" (with versions 3.0 and 2.0 also seen historically). It also sends a From header containing a contact email address (typically [email protected]).

📋 robots.txt Compliance

Moget is explicitly documented as fully compliant with the Robots Exclusion Protocol (robots.txt). Mojeek’s official crawler page states that moget respects all Disallow directives and also honors Crawl-Delay directives if specified. However, they note that moget does not support the Noindex meta tag or X-Robots-Tag HTTP headers; for exclusion, site owners must use robots.txt Disallow.

🔍 Detection Indicators

The primary identifier is the User-Agent string: "Moget/4.0" (or older "Moget/3.0", "Moget/2.0"). Additional behavioral fingerprints include a request pattern that includes a From header (e.g., [email protected]), and a tendency to request robots.txt before each crawl session. The crawler does not spoof other user-agents and consistently sends a valid, descriptive User-Agent. It also sends an Accept-Language header typically set to "en-gb,en;q=0.8".

📊 Data Usage

The data collected by moget is used exclusively to build and maintain Mojeek’s search index. Mojeek is a privacy-focused search engine that claims not to track users, sell data, or personalize results based on user history. The crawled content is processed and stored in Mojeek’s own index, which is used to provide search results to users. According to Mojeek’s privacy policy, no data from crawling is used for AI training or advertising purposes.

⚙️ Rate Limiting Policy

Because moget can be persistent and may crawl deep site structures, it is recommended to rate-limit it to prevent excessive load on web servers. The rationale is to balance Mojeek’s need to index fresh content with a site’s resource constraints. A common threshold is to allow up to 1 request per second per IP or to set a Crawl-Delay: 10 in robots.txt. Mojeek acknowledges that aggressive crawling can occur on large sites, but they encourage site owners to contact them via the email provided in the From header for custom rate adjustments.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.