🤖 Overview

searchmee! is a web crawler operated by Searchmee Ltd., a UK-based company that provides a privacy-focused search engine at searchmee.com. Its primary purpose is to index publicly accessible web pages to populate the Searchmee search index, which emphasizes user privacy by not tracking searches or storing personal data. The crawler was first introduced publicly in 2019 and has since been documented on the company’s official website and in forum posts about webmaster best practices.

🌐 Technical Behavior

The searchmee! bot follows a broad crawl pattern, starting from seed URLs submitted by webmasters or discovered through sitemaps. It sends HTTP GET requests at a configurable rate, typically between 5 and 20 requests per second per IP, but can burst higher during initial discovery. The bot primarily uses HTTP/1.1 and HTTPS, supports gzip compression, and identifies itself via a distinct User-Agent string. IP addresses used by searchmee! are allocated from ranges owned by Searchmee Ltd. and its cloud hosting providers (e.g., AWS and DigitalOcean), though the company does not publish a comprehensive IP list. The crawler respects Cache-Control and Last-Modified headers to avoid re‑downloading unchanged content, and it uses E‑Tag validation when available. It also follows nofollow and noindex meta tags as part of its crawling discipline.

📋 robots.txt Compliance

Searchmee! strictly honors robots.txt directives, as stated in its official documentation on the Searchmee webmaster portal. The bot checks the robots.txt file before each crawl and caches it for up to 24 hours. It also respects Allow and Disallow rules, including wildcard patterns. Webmasters have reported in community forums that searchmee! reliably stops crawling paths marked with “Disallow,” and there are no known incidents of the bot ignoring these instructions.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; searchmee!/1.0; +https://searchmee.com/). Some variations include a version number (e.g., “1.1”) and the bot may also add a From header with a contact email address found in its documentation. Another identifying header is X-Robots-Tag: noindex which the bot includes in its requests to signal its purpose. Behavioral fingerprints include a high frequency of HEAD requests before GET, and a preference for low‑latency responses.

📊 Data Usage

Collected content is used exclusively to build the Searchmee search index, which powers the search engine’s results. The company explicitly states that it does not sell user data or use crawled content for AI training without separate permission. The data is stored temporarily for indexing and then aggregated into a public search database that respects the noindex and nofollow signals from publishers.

⚙️ Rate Limiting Policy

Rate limiting for searchmee! is recommended because its bursty crawl pattern can temporarily overwhelm smaller websites, even though it is a legitimate bot. A threshold of 10 requests per second per IP, with a 60‑second ban window for exceeding that rate, is a common webmaster practice to protect server resources while still allowing the bot to complete its indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.