🤖 Overview

fembot is a web crawler operated by Femto Search, a privacy-respecting search engine first documented in 2019. Its primary purpose is to index publicly available web pages to populate Femto Search’s results, which emphasize user anonymity and minimal data retention. Unlike major commercial crawlers, fembot does not collect data for AI training but solely for search engine functionality.

🌐 Technical Behavior

fembot crawls using HTTP/1.1 with a default User-Agent of fembot/1.0 and respects the Crawl-Delay directive in robots.txt when set. Its requests originate from IP ranges assigned to Femto Search’s infrastructure, primarily in the 46.4.0.0/16 and 78.46.0.0/16 blocks (based on observed patterns from public server logs). The crawler typically fetches one page per second per host, but this rate can be adjusted via the Crawl-Delay directive. fembot identifies itself via the From header with an email address (e.g., [email protected]) and supports conditional GET requests with If-Modified-Since headers to reduce bandwidth usage.

📋 robots.txt Compliance

According to the official Femto Search documentation (available at https://femtosearch.com/robots), fembot strictly honors all Disallow directives in robots.txt. It also respects the Allow directive and does not override explicit exclusions. There are no known reports of fembot ignoring robots.txt rules; its compliance is verified by webmasters in community forums.

🔍 Detection Indicators

The primary User-Agent string is fembot/1.0 (case-sensitive). Additional identifying headers include X-Robots-Tag: noindex handling and a consistent Accept: text/html,application/xhtml+xml header. Behavioral fingerprints include a lack of JavaScript rendering and a fixed interval between requests unless overridden by Crawl-Delay. Server logs may show the IP range 46.4.0.0/16 as the source for most requests.

📊 Data Usage

Collected content is used exclusively for search indexing within Femto Search’s engine. The crawler stores page text, metadata, and links to build the inverted index. Femto Search’s privacy policy states that no personal data is retained and that crawled pages are not used to train machine learning models or large language models. The data is refreshed periodically to keep search results current.

⚙️ Rate Limiting Policy

Because fembot can issue rapid, sustained requests if no crawl delay is set, administrators may rate-limit it to prevent excessive load. The recommended threshold is 10 requests per second per IP; blocking is justified when the crawler exceeds this rate without respecting Crawl-Delay or when it targets non-public endpoints. This policy aligns with standard practices for legitimate but aggressive crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.