Skip to main content

Boteraser | Website and Server Security Solutions

FemtosearchBot

Search Engine User-Agent: femtosearchbot

🤖 Overview

FemtosearchBot is the web crawler operated by Femtosearch, a privacy-focused search engine developed by an independent team and first publicly documented in a 2021 blog post on femtosearch.com. Its primary purpose is to index publicly accessible web pages for the Femtoscarch search engine, which emphasizes minimal data retention and no tracking of user queries. The bot is explicitly designed to collect content only for search indexing and does not feed data into AI training or analytics platforms, as stated in the project’s official FAQ.

🌐 Technical Behavior

The bot uses a custom, lightweight HTTP client written in Go, as detailed in the open-source repository at github.com/femtoscarch/femtoscarch-crawler. It performs sequential, single-threaded crawling without parallel requests, maintaining a default delay of 10 seconds between successive requests to the same host, though this can be configured via its configuration file. The crawler supports both HTTP/1.1 and HTTP/2, and its IP ranges are drawn from a small pool of IPv4 addresses in the 185.199.108.0/24 block (hosted by Fastly) and a handful of residential-style IPs from Hetzner, as observed in server logs published by operators. According to its official documentation, the bot respects all standard HTTP protocols and does not follow redirects that change scheme from HTTP to HTTPS without explicit permission.

📋 robots.txt Compliance

FemtoSearchBot fully honors robots.txt directives, as confirmed in the project’s README on GitHub and in practical tests by webmasters. The crawler fetches and caches the robots.txt file for each domain at a minimum interval of one hour, rechecking it before every crawl session.

🔍 Detection Indicators

The bot identifies with the User-Agent string FemtoSearchBot/1.0 (compatible; https://femtoscarch.com/bot), and its requests include a custom X-FemtoSearch-Client: 1.0 header. Behavioral fingerprints include a maximum request size of 1 MB and no support for cookies or JavaScript execution, making it easily distinguishable from malicious scrapers.

📊 Data Usage

Collected data is used exclusively for building the Femtoscarch search index, which returns results from crawled pages stripped of user-identifying information. According to the project’s privacy policy, no data is shared with third parties or used for machine learning model training.

⚙️ Rate Limiting Policy

Rate limiting is advisable because, while FemtoSearchBot is legitimate and well-behaved, its fixed 10-second delay can still impose a measurable load on small sites with slow infrastructure or limited concurrent connections. Implementing a threshold-based rate limit (e.g., blocking after 50 requests per minute from the same IP) protects server resources without unfairly blocking the bot under normal conditions.

Free Bot Analysis

Is Your Site Under Bot Attack Right Now?

Find out exactly how much of your traffic is automated — and which bots are draining your bandwidth and skewing your analytics.

Run Free Bot Scan →

No credit card required  ·  Results in minutes

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓