🤖 Overview

yacybot is the web crawler component of YaCy, a fully distributed, peer-to-peer search engine first released in 2008 by the YaCy community. Unlike centralized bots, yacybot operates across thousands of independent peers running the YaCy software, each contributing crawl results to a shared, decentralized search index that is public and freely accessible. Its primary purpose is to index the World Wide Web without any central authority, ensuring censorship resistance and data privacy for users.

🌐 Technical Behavior

yacybot initiates crawls from user-configured seed URLs and expands via hyperlink discovery, using a breadth-first or depth-first strategy customizable by the peer operator. Each peer sets its own crawl frequency; default configurations typically request one page every 5–30 seconds per domain to avoid overloading servers. The bot uses HTTP/1.1 with persistent connections and supports HTTPS, IPv4, and IPv6. IP addresses are not fixed—they correspond to the public IP of the running peer, which can range globally; common ranges include residential ISP blocks and cloud provider IPs. YaCy’s network currently consists of roughly 1,000 active peers, as documented on the official statistics page at https://yacy.net/en/Stats.html. Crawling is multithreaded; a single peer may spawn up to 200 simultaneous connections (configurable) to different hosts, but domain-level throttling is enforced via its CrawlDelay setting.

📋 robots.txt Compliance

yacybot fully honors robots.txt directives, including Disallow and Crawl-Delay fields, as stated in the official YaCy documentation at https://yacy.net/en/robots.html. The bot reads the file before crawling each domain and respects both user-agent-specific rules and wildcards. In versions prior to 1.92, there was a known bug (now fixed) where Disallow paths with trailing slashes were incorrectly handled, but current releases comply strictly with the Robots Exclusion Protocol.

🔍 Detection Indicators

The primary User-Agent string is yacybot (HIT/IP-address.example.com; https://yacy.net/bot.html), varying by peer identity. Secondary identifiers may include generic YaCy versions like YACY_BOT_V1.92. Behavioral fingerprints include requesting /robots.txt as the first page, using Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8, and sending a custom From header containing the peer’s admin email if configured. Unlike many large‑scale bots, yacybot does not spoof its identity and includes a verifiable contact URL in the User-Agent.

📊 Data Usage

Data collected by yacybot is used exclusively to populate the YaCy search index, a public, decentralized database that powers the YaCy search engine. Crawled content—including page titles, text, metadata, and outbound links—is hashed and distributed across the peer-to-peer DHT network. The index is open for anyone to query via the YaCy web interface or API. No personal identifiable information is retained, and the bot does not use collected data for AI training or commercial analytics; its sole purpose is community‑driven search indexing.

⚙️ Rate Limiting Policy

Because yacybot runs on many independent peers with no central coordination, it can appear aggressive if multiple peers target the same domain simultaneously. Site administrators are advised to use throttle‑based rate limiting (e.g., limiting requests per IP per second) rather than outright blocking, as the bot is legitimate and designed to be cooperative. The YaCy community recommends setting a Crawl-Delay of 10-30 seconds in robots.txt to manage local load.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.