🤖 Overview

Surf is a general‑purpose web crawler operated by the Surf Search Engine project, an open‑source initiative hosted on GitHub under the repository github.com/surf-search/surf. Its primary purpose is to index publicly accessible web pages to build a decentralized, privacy‑focused search engine that does not track users or store personal data. First publicly documented in 2021 via the project’s official README, the bot collects text and metadata from sites that allow crawling, feeding the results into Surf’s own indexing pipeline. The project emphasizes transparency and provides a dedicated page for webmasters to review the crawler’s behavior.

🌐 Technical Behavior

Surf operates over HTTP/1.1 and HTTP/2, sending requests at a moderate rate of approximately one request per second per domain, as stated in the project’s Crawler Configuration file on GitHub. The crawler follows links recursively but limits crawl depth to a default of five levels to avoid excessive strain. IP ranges used by Surf are dynamically assigned from cloud providers such as DigitalOcean and Hetzner, with the official IP list published in the project’s ip‑ranges.txt file. The bot respects the Robots Exclusion Protocol and adheres to the robots.txt Crawl‑Delay directive. It also includes a Surf‑Request‑ID custom header for identification and debugging purposes.

📋 robots.txt Compliance

Evidence from the official GitHub repository shows that Surf strictly honors Disallow directives found in robots.txt before initiating any crawl on a domain. The crawler caches the robots.txt file for 24 hours and re‑fetches it if the response includes a Cache‑Control header indicating a shorter interval. This behavior is documented in the project’s robots.go source file.

🔍 Detection Indicators

The primary User‑Agent string for Surf is Surf/1.0, sometimes extended with “ (+https://surfsearch.org/bot)” as recorded in the project’s useragent.go file. Secondary identifying headers include From: [email protected] and Accept: text/html,application/xhtml+xml. Behavioral fingerprints include a predictable request interval of 1‑2 seconds and the absence of JavaScript rendering.

📊 Data Usage

Collected data—including page titles, meta descriptions, headings, and body text—is used exclusively to populate the Surf Search Engine’s index, which is publicly queryable at surfsearch.org. No data is sold, shared with third parties, or used for AI model training. The project’s privacy policy explicitly states that the crawler stores only the minimal content needed for search functionality and discards all logs after 48 hours.

⚙️ Rate Limiting Policy

Although Surf is a legitimate crawler, webmasters may choose to rate‑limit it using threshold‑based blocking (e.g., returning 429 Too Many Requests after 50 requests per minute) to protect server resources during high‑traffic periods, as the bot’s default crawl rate can still be aggressive on small sites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.