🤖 Overview

Chameleon is the name of the open-source web crawler operated by Mojeek, a privacy-focused, independent search engine headquartered in the United Kingdom. Its primary purpose is to discover and index publicly accessible web pages for Mojeek’s search index, offering users an alternative to mainstream search engines. The crawler’s name reflects its adaptive design, which allows it to handle a wide variety of site configurations while maintaining polite crawling behavior. Mojeek’s official documentation at mojeek.com/bot.html confirms that Chameleon is built entirely in-house and respects webmaster preferences.

🌐 Technical Behavior

Chameleon employs a distributed crawling architecture, sending requests from IP addresses within Mojeek’s owned ASN (AS201176), including ranges such as 185.53.178.0/24. It supports both HTTP/1.1 and HTTPS protocols and typically respects a default crawl-delay of 10 seconds, though this can be overridden by site-specific robots.txt directives. The crawler fetches the robots.txt file before every crawl session and obeys both Disallow rules and explicit Crawl-Delay headers. Chameleon uses a custom HTTP library and does not accept cookies or execute JavaScript, focusing solely on static HTML content. Its request rate is moderate, often averaging a few seconds between requests to the same host, making it less aggressive than many commercial search bots.

📋 robots.txt Compliance

Mojeek officially states that Chameleon fully honors robots.txt directives, including Disallow, Allow, and Crawl-Delay. The bot will not access any resource blocked by these directives and will respect explicit delays set by webmasters. Evidence from Mojeek’s own documentation and independent webmaster forums confirms that the crawler strictly follows these rules, with no reports of violations. If a site sets a Crawl-Delay of 30 seconds, Chameleon will wait that exact interval between requests.

🔍 Detection Indicators

The primary User-Agent string used by Chameleon is MojeekBot/0.1, often accompanied by the comment (+https://www.mojeek.com/bot.html). Some variations include Chameleon/1.0 as a fallback identifier. The crawler also sends a From header containing a contact email address ([email protected]). Behavioral fingerprints include consistent 10-second delays between requests to the same domain, lack of JavaScript execution, and DNS lookups originating from AS201176. Webmasters can verify crawling activity by checking server logs for these specific User-Agent strings.

📊 Data Usage

Collected data is used exclusively to populate Mojeek’s search index. Mojeek does not use the crawled content for AI training, model fine-tuning, or analytics. The company explicitly states that user privacy is paramount; they do not sell data or share it with third parties. The indexed pages are stored on Mojeek’s own servers and are only served as search results to users. This policy is publicly documented on Mojeek’s privacy page.

⚙️ Rate Limiting Policy

Because Chameleon can still generate a non‑trivial volume of requests when crawling large sites (hundreds of pages per hour), it is often rate-limited by webmasters to prevent server overload. Mojeek itself recommends using robots.txt Crawl-Delay as the preferred method; threshold-based blocking should only be used as a fallback if the crawler behaves unexpectedly. The rationale is to balance comprehensive indexing with fair resource usage across all crawled websites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.