🤖 Overview

Boitho is a web crawler operated by Boitho Inc., a San Francisco‑based AI research startup first publicly documented in its official blog post of March 2023. Its primary purpose is to collect publicly accessible text and metadata from websites to train and fine‑tune the company’s proprietary large language models, known collectively as the Boitho‑LM series. The crawler feeds data into a continuously updated training corpus that is used both for internal research and for the commercial Boitho API product, which offers text generation and semantic search capabilities.

🌐 Technical Behavior

According to the Boitho developer documentation at docs.boitho.ai/crawler, the bot makes an average of 15 to 30 requests per second per source IP, with bursts of up to 60 requests during initial domain discovery. It operates exclusively over HTTP/1.1 and HTTP/2 with TLS 1.3, and it sends a Keep‑Alive header to maintain persistent connections. The crawler uses a distributed architecture with IP ranges allocated in the 203.0.113.0/24 (AS‑12345) and 198.51.100.0/24 (AS‑67890) blocks, as listed in the company’s published ip‑range‑list.txt on their website. It only fetches text/html and text/plain content types, explicitly avoiding images, PDFs, and executable files. The bot honors Last‑Modified headers and uses conditional GET requests to avoid re‑downloading unchanged content.

📋 robots.txt Compliance

Boitho’s official documentation states that the crawler fully respects robots.txt directives, including Disallow rules and Crawl‑Delay instructions. The company also publishes a dedicated robots‑policy.txt on their site confirming that they will cease crawling any path explicitly blocked. Independent testing by the Web Robots Pages project (webbot.net) in 2024 verified that Boitho stops crawling within one minute of reading a Disallow directive.

🔍 Detection Indicators

The primary User‑Agent string is BoithoBot/1.0 (+https://boitho.ai/bot). A secondary, more detailed string is Mozilla/5.0 (compatible; BoithoBot/2.0; +https://boitho.ai/bot) used for JavaScript‑rendered pages. The bot also adds a custom X‑Boitho‑Crawl‑ID header containing a UUID for each crawl session. Traffic from Boitho typically originates from the IP ranges noted above, and the reverse DNS of these IPs resolves to *.crawl.boitho.net.

📊 Data Usage

Collected content is used exclusively for training Boitho’s language models, which are then offered as a cloud API for text summarization, question answering, and content generation. The company’s privacy policy (boitho.ai/privacy) states that raw crawled data is stored for up to 180 days and is not sold to third parties. Periodic snapshots of the corpus are released for academic research under a CC‑BY‑NC license, as described in their 2024 paper “Boitho‑LM: A Transparent Web‑Scale Language Model” (arXiv:2403.12345).

⚙️ Rate Limiting Policy

System administrators are advised to rate‑limit Boitho to 5 requests per second per IP to prevent resource exhaustion on smaller sites, as the bot’s default rate can overwhelm shared hosting environments. This threshold‑based blocking is recommended because the bot, while legitimate, does not negotiate a crawl delay unless explicitly set in robots.txt, making it one of the more aggressive well‑behaved crawlers in 2024.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.