femtosearchbot
FemtosearchBot is the web crawler operated by Femtosearch, a privacy-focused search engine developed by an independent team and first publicly documented in a 2021 blog post on femtosearch.com. Its primary purpose is to index publicly accessible web pages for the Femtoscarch search engine, which emphasizes minimal data retention and no tracking of user queries. The bot is explicitly designed to collect content only for search indexing and does not feed data into AI training or analytics platforms, as stated in the project’s official FAQ.
The bot uses a custom, lightweight HTTP client written in Go, as detailed in the open-source repository at github.com/femtoscarch/femtoscarch-crawler. It performs sequential, single-threaded crawling without parallel requests, maintaining a default delay of 10 seconds between successive requests to the same host, though this can be configured via its configuration file. The crawler supports both HTTP/1.1 and HTTP/2, and its IP ranges are drawn from a small pool of IPv4 addresses in the 185.199.108.0/24 block (hosted by Fastly) and a handful of residential-style IPs from Hetzner, as observed in server logs published by operators. According to its official documentation, the bot respects all standard HTTP protocols and does not follow redirects that change scheme from HTTP to HTTPS without explicit permission.
FemtoSearchBot fully honors robots.txt directives, as confirmed in the project’s README on GitHub and in practical tests by webmasters. The crawler fetches and caches the robots.txt file for each domain at a minimum interval of one hour, rechecking it before every crawl session.
The bot identifies with the User-Agent string FemtoSearchBot/1.0 (compatible; https://femtoscarch.com/bot), and its requests include a custom X-FemtoSearch-Client: 1.0 header. Behavioral fingerprints include a maximum request size of 1 MB and no support for cookies or JavaScript execution, making it easily distinguishable from malicious scrapers.
Collected data is used exclusively for building the Femtoscarch search index, which returns results from crawled pages stripped of user-identifying information. According to the project’s privacy policy, no data is shared with third parties or used for machine learning model training.
Rate limiting is advisable because, while FemtoSearchBot is legitimate and well-behaved, its fixed 10-second delay can still impose a measurable load on small sites with slow infrastructure or limited concurrent connections. Implementing a threshold-based rate limit (e.g., blocking after 50 requests per minute from the same IP) protects server resources without unfairly blocking the bot under normal conditions.
Free Bot Analysis
Find out exactly how much of your traffic is automated — and which bots are draining your bandwidth and skewing your analytics.
Run Free Bot Scan →No credit card required · Results in minutes
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.