🤖 Overview

minibot is an open‑source, lightweight web crawler maintained primarily by GitHub user alirezamika, hosted under the MIT license at https://github.com/alirezamika/minibot. Designed for personal website indexing, search engine prototyping, and crawling research, it is not associated with a commercial product or large‑scale AI training pipeline. The bot operates as a single‑threaded agent that respects standard web crawling ethics, making it a common choice for hobbyists and small projects.

🌐 Technical Behavior

minibot issues standard HTTP/1.1 GET requests and follows HTML anchor links recursively with a configurable depth (default: 3 levels). Requests are spaced by a fixed delay of 10 seconds by default, though this can be lowered in the source code. It originates from IP addresses in the 45.33.x.x range (DigitalOcean) and occasionally from 104.28.x.x (Cloudflare), as shown in the project’s configuration files. The bot does not support JavaScript rendering or session cookies, relying purely on static HTML parsing via Python’s BeautifulSoup library. It sends a simple Accept: text/html header and omits Accept‑Language, making it easy to distinguish from real browsers.

📋 robots.txt Compliance

minibot fully implements robots.txt parsing using Python’s urllib.robotparser, as confirmed in its source code. It checks the Disallow and Allow directives before every page request and will not crawl URLs that are blocked. The official repository includes a test suite that validates compliance with RFC 9309, ensuring the bot respects per‑path and global rules.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; minibot/1.0; +https://github.com/alirezamika/minibot), though variants like minibot/1.0 without the compatibility prefix have been observed. Behavioral fingerprints include a uniform 10‑second inter‑request delay, lack of a Referer header, and a consistent crawl order (alphabetical by URL). An optional From header may contain the operator’s email address, as documented in the README.

📊 Data Usage

Collected content is stored locally as plain text files or SQLite databases and used exclusively for personal search indexes or educational crawling experiments. The project explicitly states that data is not sold, shared, or used for AI training. No known CVEs or security advisories have been associated with minibot, as it is a trivial non‑commercial tool.

⚙️ Rate Limiting Policy

Although minibot is polite by default, its lack of crawl‑rate negotiation can cause proportional harm to small sites when many instances run simultaneously. Rate‑limiting thresholds (e.g., 50 requests per minute per IP) are recommended to prevent accidental resource exhaustion while still allowing legitimate indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.