🤖 Overview

SlySearch is a web crawler operated by SlySearch Inc., a privacy-focused search engine launched in 2022 that positions itself as an alternative to mainstream trackers. Its primary purpose is to index publicly available web pages to populate SlySearch’s own search results, emphasizing user anonymity by not storing search histories or using tracking cookies.

🌐 Technical Behavior

The SlySearch crawler, often identified as the slysearch bot, follows standard HTTP/1.1 and HTTP/2 protocols, issuing GET requests at variable intervals typically between 1 and 5 seconds per domain to avoid overloading servers. Crawl depth is limited to a maximum of 10 hops from the seed URL, and the bot respects Crawl-Delay directives in robots.txt. The IP ranges used are primarily sourced from Cloudflare and Hetzner data centers, with ASN ownership varying by region. According to SlySearch’s official documentation at https://slysearch.com/crawler, the bot checks for Last-Modified and ETag headers to minimize redundant fetches.

📋 robots.txt Compliance

Based on the publicly available crawler policy page (last updated March 2024), the SlySearch bot fully honors Disallow and Allow directives as defined in the robots.txt file per the Robots Exclusion Protocol. It also respects Crawl-Delay with a minimum delay of 1 second. The bot does not crawl pages blocked by a noindex meta tag or X-Robots-Tag HTTP header.

🔍 Detection Indicators

The primary User-Agent string is SlySearch/1.0 (+https://slysearch.com/bot), with alternate strings such as Mozilla/5.0 (compatible; SlySearch/1.0; +https://slysearch.com/bot) observed in server logs. No additional custom headers are sent, but the bot always includes an Accept: text/html,application/xhtml+xml header. Behavioral fingerprinting shows the bot does not execute JavaScript or load external resources beyond the HTML document.

📊 Data Usage

Collected page content is used exclusively to build and maintain the SlySearch search index, which is then served to end users without any ad targeting or user profiling. The company’s privacy policy explicitly states that no personal data is extracted from crawled pages, and the index is refreshed approximately every 30 days for high-traffic domains.

⚙️ Rate Limiting Policy

SlySearch is rate‑limited because its crawl volume, though polite, can still cause measurable load on smaller web servers. The recommended threshold for blocking is 50 requests per 60 seconds from a single IP, as documented on their crawler page, to protect site performance while still allowing reasonable indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.