🤖 Overview

Krowler is a web crawler operated by Kagi Inc., the company behind the privacy-focused, paid search engine Kagi (kagi.com). First publicly identified in mid-2023, Krowler is used to index publicly available web pages into Kagi’s proprietary search index, which powers its ad-free, user-centric search results. According to Kagi’s official documentation, the crawler is designed to respect publisher preferences and operates under a transparent, rate-limited policy to avoid overwhelming servers.

🌐 Technical Behavior

Krowler initiates requests from IP ranges belonging to Kagi’s infrastructure, primarily from data centers in the United States and Europe. It makes HTTP/1.1 and HTTP/2 requests with a typical frequency of 1–2 requests per second per domain, adjusting dynamically based on server response times (e.g., 503 or 429 responses cause backoff). The crawler follows redirects (up to 10 hops) and respects Cache-Control and Last-Modified headers to avoid re-indexing unchanged content. Krowler does not execute JavaScript or load external resources such as images or CSS; it fetches only the raw HTML of each page. It also sends a non-standard X-Robots-Tag header to indicate its identity, as documented in Kagi’s technical blog.

📋 robots.txt Compliance

Krowler fully obeys the robots.txt standard, including Disallow, Allow, and Crawl-Delay directives, as confirmed by Kagi’s own robots.txt at https://kagi.com/robots.txt which outlines the crawler’s behavior. The bot also honors noindex and nofollow meta tags and the X-Robots-Tag header. Kagi explicitly states that developers can block Krowler entirely by adding User-agent: Krowler with a Disallow: / line in their robots.txt.

🔍 Detection Indicators

The primary User-Agent strings are Krowler/1.0 and Krowler/1.1, often appended with a version and platform information (e.g., Krowler/1.0 (+https://kagi.com/contact)). Krowler also includes a custom header X-Kagi-Crawler: true and carries a Via header referencing its origin. Reverse DNS lookups on its IPs typically resolve to hostnames under kagi.com or krowler.kagi.com.

📊 Data Usage

Data collected by Krowler is used exclusively to build and maintain Kagi’s search index, which delivers uncensored, ad-free search results to subscribers. Kagi emphasizes that crawled content is not used for AI model training, sale to third parties, or any form of surveillance; the company’s privacy policy explicitly states that personal data is never extracted from crawled pages. The indexed data is refreshed periodically (typically every 7–14 days for active pages) to keep results current.

⚙️ Rate Limiting Policy

Krowler is rate-limited because its polite crawl rate (1–2 req/s per domain) is still aggressive enough to cause load on small or poorly configured web servers. Kagi recommends setting a Crawl-Delay: 5 in your robots.txt to further throttle requests, and threshold-based blocking (e.g., after 50 requests in 60 seconds) is considered reasonable to protect site availability without permanently banning a legitimate search engine crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.