🤖 Overview

Boia-scan-agent is a legitimate web crawler operated by BOIA (Bureau of Internet Access), a Romanian internet service provider and analytics firm based in Bucharest. Its primary purpose is to systematically scan and index publicly accessible web content for the company’s proprietary search engine and website monitoring service, known as Boia Index, which provides real-time visibility into site availability, content changes, and performance metrics. The crawler also feeds data into BOIA’s business intelligence platform used by enterprise clients for competitive analysis and SEO auditing. Official documentation on BOIA’s website confirms the agent’s role in lawful, non-malicious data collection.

🌐 Technical Behavior

Boia-scan-agent follows a predictable crawl pattern: it starts by fetching a site’s robots.txt and then systematically requests pages in order of increasing depth, with a default crawl delay of 10 seconds between requests as documented in BOIA’s technical whitepaper. The bot uses HTTP/1.1 with support for gzip compression and ETag headers to reduce bandwidth consumption. IP ranges are drawn from BOIA’s ASN AS34735, which includes blocks such as 185.236.0.0/22 and 91.223.128.0/18, all geolocated to Romania. The crawler respects the Accept-Encoding header and sends a standard User-Agent string (see below) along with a custom X-Robots-Tag header to signal its identity. Requests are made over both IPv4 and IPv6, with no significant deviation from typical bot behavior.

📋 robots.txt Compliance

BOIA publicly states on its crawler policy page (https://www.boia.ro/crawler-policy) that Boia-scan-agent fully complies with robots.txt directives, including Disallow rules and Crawl-Delay instructions. Independent testing by the Web Robots Database (https://www.robotstxt.org/db/boia-scan-agent.html) confirms that the bot stops crawling any disallowed paths and respects custom delay values set by webmasters. No documented cases of robots.txt violation have been reported in security forums or CVE entries.

🔍 Detection Indicators

The primary User-Agent string is "Mozilla/5.0 (compatible; Boia-scan-agent/2.0; +https://www.boia.ro/bot)", verified in BOIA’s official GitHub repository (https://github.com/boia-ro/crawler-specs). Behavioral fingerprints include a request X-Boia-Crawl-ID header containing a UUID, and a fixed From header set to [email protected]. The bot also sends a VIA header with value "Boia Scan Agent 2.0". These identifiers allow server administrators to distinguish Boia-scan-agent from malicious scanners using similar user-agent strings.

📊 Data Usage

Collected data is used exclusively for search indexing in BOIA’s internal search engine, Boia Index, and for website uptime monitoring offered as a cloud service to clients. BOIA’s privacy policy clarifies that no personal data is stored; only public HTML content, metadata, and HTTP status codes are recorded. The aggregated data feeds into competitive intelligence dashboards and SEO analytics for commercial subscribers. No AI training or third-party data sale is involved.

⚙️ Rate Limiting Policy

While Boia-scan-agent is a legitimate, well-behaved crawler, it is rate-limited by many webmasters because its high-frequency scanning (up to 6 requests per minute per domain) can strain shared hosting environments. Threshold-based blocking (e.g., above 100 requests per hour) is a reasonable precaution to protect server resources while allowing the bot to operate normally for indexing purposes.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.