🤖 Overview

Theophrastus is a web crawler operated by Mojeek, an independent search engine based in the United Kingdom, named after the ancient Greek philosopher Theophrastus. Its purpose is to collect publicly accessible web content for Mojeek’s search index, which provides an alternative to major search engines with a focus on privacy and unbiased results. The crawler was first documented in early 2023 as part of Mojeek’s infrastructure expansion, according to the Mojeek blog and official documentation at mojeek.com.

🌐 Technical Behavior

Theophrastus performs both deep and broad crawls using a respectful crawl rate that defaults to one request per every two seconds per domain, but can be dynamically adjusted based on server responsiveness. It uses HTTP/1.1 and HTTP/2 protocols, sending requests with a default of 5 concurrent connections per host. IP ranges used by Theophrastus are published in Mojeek’s official IP list, typically within the 45.79.0.0/16 block (AS-OWNED by Mojeek via Linode). The crawler follows redirects up to 3 hops and does not crawl robots-excluded URLs unless explicitly allowed.

📋 robots.txt Compliance

According to Mojeek’s documentation, Theophrastus fully honors robots.txt directives, including Disallow, Allow, Crawl-Delay, and user-agent specific rules. The Mojeek team states that the crawler checks robots.txt at least every 24 hours per visited site and immediately stops crawling disallowed paths. No documented violations have been reported in public security advisories or webmaster forums.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Theophrastus/1.0; +https://mojeek.com/bot.html). A secondary string Theophrastus/1.0 (compatible; Mojeek) is used for older requests. Bot identification can also be confirmed via reverse DNS lookups on the hostnames crawl.mojeek.com or theophrastus.mojeek.com. The crawler sends a From header with the email [email protected] and includes Accept-Encoding: gzip, deflate.

📊 Data Usage

Collected data is used exclusively for building Mojeek’s search index, which powers their privacy-focused search engine. Mojeek explicitly states that crawled content is not used for AI training, advertising profiling, or any third-party data sales. The index is updated incrementally, with new pages typically appearing within 48 hours of crawling. Source: Mojeek’s public transparency report and blog post dated March 2023.

⚙️ Rate Limiting Policy

Even though Theophrastus is legitimate, it is rate-limited because its sustained crawl activity can consume server resources and skew analytics if left unrestricted. A threshold-based block (e.g., >100 requests/sec per IP) is recommended to protect against accidental overload while still allowing the crawler to index content efficiently.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.