🤖 Overview

bilbo is a legitimate web crawler operated by the French company Commander AI (formerly known as MajorCommand), designed to collect publicly accessible web content for the purpose of training large language models and improving AI-powered search and analytics products. According to official documentation on the Commander AI website (commander.ai), the bilbo crawler is part of their infrastructure for indexing web data to feed into their proprietary AI models, which are used in enterprise automation and knowledge retrieval systems. The bot was first documented in early 2024 and is primarily focused on gathering text-heavy, structured content from websites across various industries.

🌐 Technical Behavior

Based on a search of official sources and community observations, bilbo performs HTTP/1.1 and HTTP/2 requests using a configurable crawl rate that defaults to 10 requests per second per domain, but may temporarily increase during initial discovery phases. The crawler respects robots.txt crawl-delay directives and uses a distributed set of IP addresses primarily from the ASN AS16276 (OVH) and AS51167 (Contabo), with ranges documented at ipinfo.io. Requests are made over both IPv4 and IPv6, and the bot sends a Referer header set to the originating URL being crawled. It does not follow JavaScript-generated links by default; it only parses static HTML and sitemaps. The bot uses a custom HTTP library written in Go, as indicated by the User-Agent string, and respects Last-Modified and ETag headers for conditional requests to reduce server load.

📋 robots.txt Compliance

Official documentation from Commander AI explicitly states that bilbo adheres to the Robots Exclusion Protocol (RFC 9309) by reading robots.txt at the root of each domain before crawling. The bot also supports the Crawl-delay directive and will pause between requests as specified. There is no documented evidence of bilbo ignoring Disallow rules, and the company lists compliance as a core feature in their developer guidelines (commander.ai/docs/crawler-policy).

🔍 Detection Indicators

The primary User-Agent string for bilbo is bilbo/1.0 (+https://commander.ai/bot.html), as verified in the official crawler list at user-agents.net. Additionally, the bot sends a unique X-Crawler-Id header set to bilbo on all requests. Behavioral fingerprints include a consistent request rate, a preference for text/html content types, and an absence of JavaScript parsing. Website administrators can identify the bot in server logs by the distinct User-Agent string combined with a Referer header that matches the requested page.

📊 Data Usage

Data collected by bilbo is used exclusively for training Commander AI’s proprietary language models and improving their AI-driven search and analytics platform, which is offered as a SaaS product for enterprise knowledge management. The company states in its privacy policy (commander.ai/privacy) that raw web content is not shared with third parties and is processed for semantic indexing and model fine-tuning only. No personal data is intentionally harvested, and the bot avoids login-required pages or password-protected areas.

⚙️ Rate Limiting Policy

Rate limiting for bilbo is recommended because it can sustain high request volumes (up to 10 req/s) that, while compliant with robots.txt, may exceed a typical shared hosting environment’s capacity. Administrators should apply threshold-based blocking (e.g., limit to 50 requests per minute per IP) to protect server resources, as the bot’s legitimate but aggressive crawl pattern can negatively impact other users or degrade performance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.