🤖 Overview

BoardReader is a specialized web crawler operated by BoardReader LLC, a company founded in 2005 that operates the vertical search engine boardreader.com dedicated exclusively to indexing internet forums, bulletin boards, and discussion communities. Unlike general-purpose search engines, BoardReader focuses solely on threaded conversations and user-generated content from millions of forums worldwide, making them searchable via its portal. The bot’s purpose is to provide a way for users to find forum discussions on specific topics.

🌐 Technical Behavior

The BoardReader bot performs targeted crawling of forum URLs, following patterns such as pagination links (`page=2`), thread views (`showthread.php`), and forum listings (`forumdisplay.php`). According to its official documentation, the bot requests pages at a rate of approximately one request per 1–2 seconds per domain, but it adjusts based on server response times and respects `Crawl-Delay` directives in robots.txt. Its IP addresses originate from ranges allocated to SoftLayer (AS36351) and Microsoft Azure (AS8075), including `208.94.148.0/24`. The bot uses HTTP/1.1 with gzip compression and identifies itself via the User-Agent string. It does not execute JavaScript and only fetches static HTML content.

📋 robots.txt Compliance

The BoardReader bot explicitly honors robots.txt Disallow directives, as confirmed in its official FAQ and observed in real-world crawls. Site owners can block it entirely by adding `User-agent: BoardReader` followed by `Disallow: /`. The bot also respects `Allow` and `Crawl-Delay` instructions. There is no documented evidence of the bot ignoring or bypassing robots.txt rules.

🔍 Detection Indicators

The primary User-Agent string is "BoardReader/1.0 (http://boardreader.com/)" or "BoardReader/1.0 (https://boardreader.com/)". A variant `BoardReader-1.0` may also appear. Behavioral fingerprints include requests to forum-specific paths (e.g., `/showthread.php`, `/viewtopic.php`) and a lack of `Referer` or `Accept` headers typical of modern browsers. The bot typically does not set cookies or session identifiers.

📊 Data Usage

Collected data is used exclusively to populate the BoardReader search index, enabling users to search across thousands of forums by keyword, view snippets, and link back to original threads. The company states that data is not used for AI training, language model development, or sold to third parties; it remains within the BoardReader platform for public search purposes only.

⚙️ Rate Limiting Policy

BoardReader is rate-limited because its focused crawling on forum sites—which often have limited server resources—can generate burst traffic. A threshold of 5–10 requests per minute per IP is recommended to prevent performance degradation while still allowing legitimate indexing of forum content.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.