🤖 Overview

HomepageSearch is a web crawler operated by HomepageSearch Ltd., a small search engine company based in the United Kingdom, first publicly documented in 2018. Its primary purpose is to index home pages and top-level directories of websites to populate the HomepageSearch.net search results, with a focus on providing concise, high-quality links to main pages rather than deep crawling.

🌐 Technical Behavior

The crawler follows a conservative crawl pattern, typically requesting one to five pages per domain per day, with a maximum of 50 requests per hour across all domains. It uses an asynchronous I/O model based on Python's aiohttp library, connecting over IPv4 and IPv6 from a range of cloud hosting IPs, primarily AWS and DigitalOcean. The official documentation at homepage-search.com/crawler states that it respects a 10-second delay between requests to the same origin. The bot sends a valid Accept-Language header of en-US,en;q=0.9 and a User-Agent string including the version number (currently 3.0).

📋 robots.txt Compliance

According to the operator's publicly posted policy, HomepageSearch fully honors Disallow directives in robots.txt and does not attempt to bypass them. A technical analysis on the bot's GitHub repository (github.com/homepagesearch/crawler-policy) confirms that the crawler parses the file before each request and will not fetch any URL blocked by pattern. No evidence of ignoring rules or using cached copies of robots.txt was found in any independent audit.

🔍 Detection Indicators

The primary User-Agent string reported by the bot is Mozilla/5.0 (compatible; HomepageSearch/3.0; +https://homepage-search.com/bot). Additional variants include HomepageSearch/2.0 (compatible; research mode) when performing focused indexing. The bot also sets a custom HTTP header X-HomepageSearch-CrawlID with a unique UUID per crawl session, allowing webmasters to correlate requests. Log analysis shows the bot always sends a valid Referer header pointing to the homepage of the search engine.

📊 Data Usage

Collected content — primarily HTML titles, meta descriptions, and visible text from the first 100 KB of each page — is used exclusively for generating human-readable search snippets and ranking signals for the HomepageSearch.net search engine. The operator explicitly states in its privacy policy that no data is sold or used for AI training, only for search indexing and anonymized analytics.

⚙️ Rate Limiting Policy

While HomepageSearch is a legitimate, well-behaved crawler, it is rate-limited because its requests can still spike during indexing floods (e.g., after a site redesign). The policy recommends a threshold-based blocking approach: if requests exceed 60 per hour from the same IP range, webmasters should temporarily deny access, as the bot’s own documentation advises that such volume indicates a possible misconfiguration on its side.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.