🤖 Overview

Kagi-Fetcher is the official web crawler operated by Kagi Inc., the privacy-focused search engine launched in 2020 by Vladimir Prelovac. The bot indexes web pages exclusively for Kagi’s own search index, which is built on a custom ranking algorithm that prioritizes user privacy and ad-free results. Unlike many search engine bots, Kagi-Fetcher is not used for any AI model training or data reselling; its sole purpose is to provide fresh, relevant search results for Kagi’s paid subscribers.

🌐 Technical Behavior

Kagi-Fetcher crawls from a set of IP addresses registered to Kagi’s infrastructure, primarily in the United States and Europe. The crawler respects a default crawl delay of 5 seconds per host, but this can be adjusted via the Crawl-Delay directive in robots.txt. Requests are made over HTTP/1.1 and HTTP/2 using the GET method, and the bot explicitly identifies itself with the User-Agent string Kagi-Fetcher (+https://kagi.com/support/bot). It also includes a From header containing a contact email ([email protected]). Kagi-Fetcher does not follow redirects that lead to JavaScript-dependent pages and avoids crawling known spam or low-quality domains based on Kagi’s internal blacklist.

📋 robots.txt Compliance

According to Kagi’s official documentation published at https://kagi.com/support/bot, Kagi-Fetcher fully honors robots.txt Disallow directives and respects the Crawl-Delay setting. The bot also supports the Allow directive for selectively permitting crawling of subdirectories. Kagi explicitly states that site owners can block the bot entirely by adding User-agent: Kagi-Fetcher followed by Disallow: / to their robots.txt file.

🔍 Detection Indicators

The primary detection method is the User-Agent string Kagi-Fetcher with the comment URL +https://kagi.com/support/bot. Additionally, the bot sends a From HTTP header with [email protected]. Reverse DNS lookups on its source IPs resolve to hostnames under kagi.com. The bot’s request pattern typically shows a consistent 5‑second gap between requests to the same host, unless overridden by Crawl-Delay.

📊 Data Usage

Data collected by Kagi-Fetcher is used exclusively to build and maintain Kagi’s search index. The index does not feed into any generative AI product or training dataset. Kagi’s privacy policy states that raw crawled content is stored temporarily (up to 30 days) for indexing and is not shared with third parties. The company also offers a Kagi Privacy Pass feature that lets users bypass tracking while searching.

⚙️ Rate Limiting Policy

While Kagi-Fetcher is legitimate and respectful, it can still generate significant traffic on high-traffic sites due to the nature of comprehensive indexing. Rate limiting with a threshold of e.g. 200 requests per minute is recommended to prevent unintended load on application resources, especially if the site serves dynamic content that triggers expensive database queries. Kagi itself recommends that site owners use the Crawl-Delay directive as the primary method for controlling crawl intensity.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.