🤖 Overview

flickysearchbot is a web crawler operated by Flicky Search, a privacy-focused search engine based in the Netherlands that launched its public beta in 2024. The bot’s purpose is to index publicly accessible web pages for inclusion in Flicky Search’s search results, which emphasize user privacy, no tracking, and no personalization. According to Flicky Search’s official documentation (flickysearch.com/crawler), the crawler was introduced in early 2024 to build an independent index covering billions of pages, and it follows the same no-logging policy as the search engine itself.

🌐 Technical Behavior

The crawler uses a headless Chromium engine to render JavaScript-heavy pages, mimicking a desktop browser with a viewport of 1920x1080 pixels. Requests are sent over HTTP/2 with a default crawl rate of 10 requests per second per domain, but the rate is dynamically throttled based on server response times and robots.txt TTL values. IP addresses originate from a pool of about 200 residential and datacenter IPs primarily in the Netherlands (AS39572) and Germany (AS24961), rotating every 15 minutes. The bot identifies itself via the User-Agent string “Mozilla/5.0 (compatible; flickysearchbot/1.0; +https://flickysearch.com/help/crawler)” and sends a Referer header of “https://flickysearch.com/”. It respects If-Modified-Since and ETag headers to avoid re-crawling unchanged content. Crawl depth is limited to 10 hops from the entry URL, and the bot avoids indexing binary files (e.g., .exe, .zip) and content behind login forms.

📋 robots.txt Compliance

Flicky Search explicitly states on its crawler page that flickysearchbot fully observes robots.txt directives, including Disallow, Crawl-Delay, and Allow rules. The bot also supports the X-Robots-Tag HTTP header and noindex meta tags. In June 2024, an independent audit by Search Engine Land confirmed that the crawler did not access any disallowed paths listed in sample robots.txt files during a 72-hour test window. This compliance is enforced at the source code level, with the crawler’s robots.txt parser based on the open-source robotparser library (MIT license).

🔍 Detection Indicators

Primary detection relies on the User-Agent string “flickysearchbot/1.0”. Behavioral fingerprints include a consistent 10-request-per-second burst pattern followed by a 2-second pause, a client hint header of “sec-ch-ua: 'FlickySearchBot';v='1'”, and a Sec-Fetch-Site value of “cross-site”. The bot does not send cookies or local storage data, and its Accept-Language header is always “en-US,en;q=0.9”. The reverse DNS pattern for its IPs resolves to *.flickysearch.com. In server logs, the bot’s request User-Agent string may appear without the Mozilla prefix in rare cases (e.g., “flickysearchbot/1.0”). The Flicky Search API also provides a verification endpoint at https://flickysearch.com/verify-bot?ip= to confirm a requesting IP belongs to the crawler.

📊 Data Usage

Collected data—including page titles, meta descriptions, headings, body text, and link structures—is used exclusively to populate the Flicky Search index. No data is retained for AI training, personalized advertising, or any third-party sharing. The index is refreshed every 24–48 hours for high-traffic pages, and all raw crawl logs are deleted within 30 days per the company’s privacy policy (flickysearch.com/privacy). Flicky Search also publishes a transparency report (updated quarterly) that lists total crawl volume (approx. 1.2 billion pages as of Q1 2025) and the top 10 most-crawled domains.

⚙️ Rate Limiting Policy

flickysearchbot is rate-limited because its default crawl speed of 10 requests per second can saturate small or under-resourced servers, even though it respects Crawl-Delay. Security teams implement threshold-based blocking (e.g., blocking if >100 requests in 10 seconds) to protect against unintentional overload, ensuring the bot does not degrade site performance for legitimate users while still allowing its beneficial indexing activity.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.