🤖 Overview

Searxbot is the web crawler component of the open‑source Searx metasearch engine, developed by the community‑maintained project hosted at github.com/searx/searx. Unlike conventional search‑engine crawlers that build a central index, Searxbot is used by individual Searx instances to fetch live search results from upstream providers (e.g., Google, Bing, DuckDuckGo) on behalf of users who prefer privacy‑preserving, self‑hosted search. The bot operates only when an instance administrator configures it to pre‑fetch or cache results; it does not indiscriminately crawl the web. Its primary purpose is to reduce latency for commonly requested queries by maintaining a local cache, while respecting the terms of service of the upstream sources it queries.

🌐 Technical Behavior

Searxbot does not perform broad, autonomous crawling like Googlebot. Instead, it issues targeted HTTP(S) requests to specific search engine APIs or HTML endpoints, mirroring user search queries. The request frequency is governed by the instance configuration in settings.yml where administrators set max_request_timeout and request_timeout values. Typical deployments limit concurrent requests to 2–5 per upstream engine to avoid triggering CAPTCHAs or rate limits. IP ranges are not fixed—they correspond to the IP address of the self‑hosted server running the Searx instance. The bot supports both HTTP/1.1 and HTTP/2 depending on the server stack. It respects robots.txt of the upstream engines only indirectly, as those engines enforce their own policies via API keys or user‑agent blocking. Searxbot does not parse or follow links; it only retrieves search result pages.

📋 robots.txt Compliance

Searxbot is not a web‑wide crawler, so traditional robots.txt compliance is largely irrelevant. However, when it makes HTTP GET requests to third‑party search engines, those engines’ robots.txt files may be checked by the Searx instance if the administrator configures a user‑agent string that mimics a real browser. By default, Searx sends a generic user‑agent (often "Mozilla/5.0 ...") to avoid being blocked, which means it effectively bypasses robots.txt disallow directives—a design choice that has drawn criticism from some engine operators. The Searx documentation explicitly warns that users must assume legal responsibility for their instance’s behavior.

🔍 Detection Indicators

Because each Searx instance can set arbitrary user‑agents, there is no single fixed string. Common configurations send a standard browser user‑agent (e.g., "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ..."). However, Searxbot can be identified by its request patterns: consecutive requests to different search engines within milliseconds, absence of JavaScript execution, and consistent Accept‑Language headers (often `en‑US,en;q=0.5`). Webmasters may also see a client IP from a known cloud hosting provider (e.g., DigitalOcean, Linode) that hosts a public Searx instance.

📊 Data Usage

The data retrieved by Searxbot is used exclusively to serve search results to end users of that particular Searx instance. No data is shared with the Searx project central servers; caching is local and entirely under the administrator’s control. The bot does not collect or store personal information. Some instances may log queries for troubleshooting, but the default setting disables logging. The project’s privacy policy (searx.github.io/searx/privacy.html) emphasizes that all queries are anonymized by default.

⚙️ Rate Limiting Policy

Rate limiting Searxbot is recommended because aggressive configurations can overload upstream search engines and violate their terms of service. Administrators should apply per‑IP‑based throttle limits (e.g., 1 request per 5 seconds per engine) and block any instance that exhibits abusive patterns, as documented in the Searx wiki on GitHub (github.com/searx/searx/wiki/FAQ).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.