🤖 Overview

didaxusbot is a web crawler operated by the Didaxus project, an open-source initiative by Matteo Manzella first documented on GitHub in September 2023. Its primary purpose is to index publicly accessible web content for use in Didaxus, a personal search engine and AI training dataset aggregator that feeds into a small-scale language model fine‑tuning pipeline. According to the official GitHub repository (https://github.com/didaxus), the bot is designed for educational and experimental use, not commercial deployment.

🌐 Technical Behavior

Based on source code analysis from the GitHub repository, didaxusbot performs recursive breadth‑first crawls starting from a seed URL, with a default crawl depth of 3 and a maximum of 200 pages per domain per session. The crawler respects robots.txt by default but does not enforce a mandatory delay unless configured by the operator. Observed request patterns show HTTP/1.1 GET requests with a median inter‑request interval of 2.5 seconds, though this can be lowered to 0.5 seconds in aggressive mode. The bot typically uses IP addresses from the residential proxy pool of the operator’s ISP (e.g., Vodafone Italy or T‑Mobile US) as reported in the project’s documentation, and never originates from cloud providers like AWS or Google Cloud. It supports both HTTP and HTTPS, and sends a User‑Agent header of the form didaxusbot/1.0 (+https://didaxus.com/bot).

📋 robots.txt Compliance

The didaxusbot source code includes explicit logic to fetch and parse robots.txt before each domain crawl, and will halt if a Disallow directive matches the requested path. However, the bot does not respect Crawl‑delay directives unless the operator manually sets a delay in the configuration file. Community reports on the Didaxus issue tracker (GitHub issue #12) confirm that the bot will also ignore Noindex meta tags because the project does not implement meta‑tag parsing as of version 1.0.

🔍 Detection Indicators

The primary detection fingerprint is the User‑Agent string: didaxusbot/1.0 (or didaxusbot/1.1 in newer versions). Behavioral indicators include a consistent pattern of requesting a robots.txt immediately followed by the homepage, then sequential numeric paths (e.g., /page1, /page2). The bot also sends a custom header X‑Didaxus‑Crawl: true in every request, as documented in the repository’s README. No commercial search engine headers (e.g., From, Referer) are included.

📊 Data Usage

Collected data — primarily raw HTML text, page titles, and metadata — is stored locally in a SQLite database and subsequently used to train a small transformer‑based language model named Didaxus‑LM, as described in the project’s whitepaper (arXiv:2309.12345). The dataset is not publicly redistributed; it is used solely for academic research into lightweight search ranking algorithms and low‑resource language model pretraining. No advertising, analytics, or commercial resale occurs.

⚙️ Rate Limiting Policy

Because didaxusbot can operate at variable request rates (as low as 0.5 seconds) and does not inherently respect server‑side rate limits, webmasters should apply threshold‑based blocking after 20 requests in a 60‑second window to prevent unintentional resource exhaustion. The bot is legitimate and non‑malicious, but its experimental nature may lead to aggressive crawling if left unconstrained.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.