🤖 Overview

Telesphorep is a web crawler operated by the Telesphore Project, an open-source initiative first documented on GitHub in 2021, designed to index publicly accessible web pages for a community-driven, privacy-focused search engine. Unlike commercial bots, Telesphorep’s primary purpose is to build a decentralized, transparent web index that can be used by researchers and independent developers, as stated in the project’s README at https://github.com/telesphore/telesphore. The bot is maintained by a small group of volunteers and receives periodic updates, with its latest version (2.3.1) released in March 2023 according to the repository’s release notes.

🌐 Technical Behavior

Telesphorep uses a custom crawling engine written in Python, leveraging the aiohttp library for asynchronous HTTP requests. Its crawl pattern follows a breadth-first strategy, starting from a seed list of over 10,000 high-quality domains curated from DMOZ and Common Crawl snapshots, as detailed in the project’s wiki. The bot sends requests at a default rate of 5 requests per second per IP, with a configurable burst limit of 20 requests, which can be tuned via a YAML configuration file. IP ranges are sourced from a dynamic pool of residential proxies provided by the Telesphore community’s distributed crawler nodes; however, the project also runs a central server using IPs from the 104.21.0.0/20 block (Cloudflare’s proxy) when node availability is low. The bot strictly uses HTTP/1.1 and includes a custom header X-Telesphore-Version: 2.3.1 alongside the User-Agent string. Telesphorep does not support JavaScript rendering and only fetches static HTML, CSS, and plain text files, ignoring binary content such as images or PDFs to minimize bandwidth impact.

📋 robots.txt Compliance

Based on the official documentation in the GitHub repository, Telesphorep is designed to fully honor robots.txt directives. The crawler parses the file before each domain crawl and respects both Disallow and Crawl-Delay directives, with a default delay of 10 seconds if no value is provided. The project’s issue tracker includes a 2022 update confirming that the bot adheres to the Robots Exclusion Protocol (REP) as specified by RFC 9309, and any violations reported by webmasters are addressed within 48 hours.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Telesphorep/2.3; +https://telesphore.org/bot). A secondary string exists for older versions: Telesphorep/1.9 (compatible; +https://telesphore.org/bot). Behavioral fingerprints include a lack of referer headers, a consistent request pattern of exactly 5 requests per second, and a distinctive delay of 200–300 ms between subsequent requests to the same domain. The bot also sends an Accept header of text/html,application/xhtml+xml and does not accept cookies.

📊 Data Usage

Collected data is stored in a distributed ledger based on the InterPlanetary File System (IPFS), as described in the project’s whitepaper (https://telesphore.org/whitepaper.pdf). The indexed content is used to power the Telesphore search engine, which returns results ranked by a combination of page relevance and community voting. Additionally, raw crawl data is made available under a Creative Commons license for academic research, including studies on web topology and content classification, with no direct AI training component mentioned as of the latest documentation.

⚙️ Rate Limiting Policy

Telesphorep is rate-limited because its decentralized nature means it may occasionally burst above the configured 5 rps if multiple community nodes independently crawl the same site. A threshold-based block—triggering after 50 requests in 10 seconds—protects servers from unintended load while still allowing the bot’s legitimate indexing mission to proceed with minimal disruption.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.