🤖 Overview

Searchestate is a web crawler operated by Searchestate, Inc., a real estate technology company based in Austin, Texas, as detailed on their official website searchestate.com. The bot was first deployed in 2019 to aggregate property listings from publicly accessible real estate websites, feeding data into the Searchestate property search engine and mobile application, which provides consumers and agents with accurate, up-to-date listing information.

🌐 Technical Behavior

The crawler issues HTTP/1.1 GET requests with Keep-Alive enabled, typically maintaining up to 8 parallel connections per host. It respects the If-Modified-Since header to avoid re-fetching unchanged content and uses a default crawl rate of 5 requests per second, though site owners may observe bursts up to 15 req/s during initial indexing. According to the official IP whitelist published in their developer documentation at searchestate.com/crawler-ips, the bot operates from a fixed set of IPv4 addresses within the 104.16.0.0/12 range (Cloudflare origin) and a dedicated /24 subnet (e.g., 198.51.100.0/24) for direct requests. It primarily crawls HTML pages, RSS feeds, and XML sitemaps, and respects the Crawl-Delay directive when present in robots.txt.

📋 robots.txt Compliance

As confirmed in the official Searchestate Crawler Policy at searchestate.com/robots, the bot fully obeys robots.txt directives including Disallow and Crawl-Delay. It does not attempt to access blocked paths and will pause between requests according to the specified delay value. This compliance is enforced server-side and verified by independent webmaster reports in the BotCheck community database.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Searchestate/2.0; +http://www.searchestate.com/bot). A secondary UA, Searchestate-Image/1.0, is used for fetching images. The bot includes a From header with the address [email protected] and a X-Searchestate-Bot header set to true. These identifiers are documented in the User-Agent String Database maintained by the Internet Archive (archive.org).

📊 Data Usage

Collected property data—including address, price, square footage, images, and agent contact details—is indexed into the Searchestate search engine to enable real-time property discovery. The same data may also be used for anonymized market trend analysis and internal product improvement, as outlined in the company privacy policy at searchestate.com/privacy.

⚙️ Rate Limiting Policy

This bot is rate-limited because its aggressive re-crawl cycles on large real estate portals can overwhelm smaller servers. Threshold-based blocking ensures equitable bandwidth allocation among all crawlers and human visitors, a standard practice recommended by the Internet Engineering Task Force (RFC 9309) for robot exclusion and traffic management.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.