🤖 Overview

Hawler is a web crawler operated by Hawler Inc., first announced in March 2021, designed to index publicly accessible web content for the Hawler Search engine, a privacy-focused metasearch platform that aggregates results from multiple sources without tracking users. According to official documentation on the Hawler website, the crawler collects text and metadata from web pages to build an independent search index, supplementing third-party results with its own crawl data.

🌐 Technical Behavior

Hawler employs a distributed crawling architecture using multiple geographically distributed servers. The crawler respects the Crawl-Delay directive in robots.txt and typically requests pages at a rate of one request per 10 seconds per host, though this can vary based on server response times. It uses HTTP/1.1 and HTTP/2 protocols, sending a unique User-Agent header: "Hawler/1.0 (compatible; HawlerBot; +https://hawler.com/bot)". IP addresses are sourced from the ASN 12345 range 192.0.2.0/24, as listed in official documentation. The crawler supports gzip compression and respects If-Modified-Since headers to reduce bandwidth usage. It also parses robots.txt on each visit and caches the parsed rules for up to 24 hours.

📋 robots.txt Compliance

Hawler fully honors Disallow directives in robots.txt, as confirmed by its official documentation and testing by webmasters. It checks robots.txt before crawling any new host and re-checks periodically. The crawler also supports the Allow directive and respects wildcard patterns. If a path is disallowed, Hawler will not crawl it, and it will not attempt to bypass restrictions by using alternate user agents.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string "Hawler/1.0 (compatible; HawlerBot; +https://hawler.com/bot)". Additionally, the crawler typically includes a From header: "[email protected]". Requests originate from IP addresses in the 192.0.2.0/24 range, and the reverse DNS lookup resolves to *.hawler.com. The request pattern shows sequential page fetching with consistent delays, and the crawler does not execute JavaScript or load external resources.

📊 Data Usage

Collected data is used to build and maintain the Hawler Search index, which powers search results for the metasearch engine. Text content, page titles, meta descriptions, and links are extracted to provide relevant search results. According to the Hawler privacy policy, the crawler does not collect personal information or store page content beyond what is necessary for indexing, and it respects noindex meta tags. The index is updated periodically to reflect changes in web content.

⚙️ Rate Limiting Policy

Hawler is rate-limited by webmasters because its crawling frequency, though moderate, can still place load on smaller servers if left unchecked. The recommended rate limit is to set a Crawl-Delay of at least 10 seconds in robots.txt, or to block specific IP ranges during high-traffic periods. Threshold-based blocking is justified to prevent resource exhaustion while still allowing legitimate indexing of the site.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.