🤖 Overview

YaCy (often referred to as yacy.net) is a decentralized peer-to-peer search engine maintained by the YaCy community and the non-profit YaCy Foundation. Originally created by Michael Christen in 2003, YaCy operates as a free, open-source software project under the GPLv2 license, with its source code hosted on GitHub (github.com/yacy/yacy_search_server). Unlike centralized search engines, each peer independently crawls web pages and shares the resulting index with the network, distributing the workload across thousands of volunteer-operated nodes. The project’s official website at yacy.net provides documentation, downloads, and community forums.

🌐 Technical Behavior

Each YaCy peer uses a Java-based crawler (YaCy Search Server) that respects standard web crawling protocols. Crawling is initiated by user-defined seeds or through the network’s shared crawl requests. The default crawl depth is configurable, typically 3–5 hops, with a crawl delay of at least 500 milliseconds between requests to the same domain (adjustable via the YaCy administrative interface). Peers identify themselves primarily via the User-Agent string YaCy/1.9x (YACY_USER_AGENT) where the version number varies (e.g., 1.92 as of late 2023). IP ranges are unpredictable since each peer runs on a volunteer’s machine or server; common hosting IPs include residential and cloud providers. The crawler uses HTTP/1.1 and performs both GET and HEAD requests. It also supports HTTPS and obey the robots.txt instructions. The network’s distributed nature means request frequency from a single IP is generally low (typically a few requests per minute per domain), but aggregate requests from the entire network can be higher.

📋 robots.txt Compliance

YaCy is explicitly designed to honor robots.txt rules. The official documentation at yacy.net/en/robots.html states that the crawler reads the Disallow directives and respects the Crawl-delay directive. By default, YaCy will not crawl URLs that are disallowed. However, individual peer operators may modify the crawler’s code or configuration to ignore robots.txt, though this violates the project’s guidelines. The project also supports the nofollow and noindex meta tags in HTML pages.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string pattern: YaCy/ followed by a version number (e.g., YaCy/1.92). Additional fingerprints include the use of the Accept header text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 and the Accept-Language header set to en or the peer’s locale. The crawler may also include the header X-YaCy-Crawl: yes in some versions. Behavioral fingerprints include low request rates per IP (sub-10 requests per minute per domain) and the absence of JavaScript rendering. The project’s source code (GitHub) allows identification of default timeout values and crawl policies.

📊 Data Usage

Collected data is used to build a shared, distributed search index for the YaCy peer-to-peer network. Each peer stores crawled content in its own database (using Apache Solr/Lucene), and search queries are forwarded to multiple peers that return matching results. The data is not used for commercial AI training or analytics; it solely powers the decentralized search engine. The project emphasizes privacy — no logs are centrally collected, and users can operate their own peer to control their own crawling and indexing.

⚙️ Rate Limiting Policy

YaCy is a legitimate, non-malicious crawler, but it may be rate-limited to prevent excessive load on servers, especially when the network has many peers crawling the same domain simultaneously. A reasonable threshold (e.g., 20 requests per minute per IP) is recommended by the community to avoid overwhelming target websites. The rate limit protects the server while still allowing YaCy to function as intended for search indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.