🤖 Overview

The Botswana crawler is a legitimate web archiving agent operated by the University of Botswana Library Services in partnership with the Botswana National Archives and Records Services. Its primary purpose is to harvest publicly accessible web content from the .bw country-code top-level domain and related sites to preserve Botswana’s digital heritage and support academic research. First documented in a 2019 paper titled “Web Archiving in Botswana: Challenges and Opportunities” published in the Journal of the South African Society of Archivists, the crawler is part of a broader national digital preservation initiative modeled on the Internet Archive’s methodology.

🌐 Technical Behavior

The Botswana crawler operates on a weekly crawl cycle, targeting approximately 15,000 registered .bw domains and a curated list of government, educational, and cultural websites. It uses HTTP/1.1 and HTTPS with support for TLS 1.2, sending requests sequentially rather than in parallel to avoid overloading small servers. The crawler originates from a fixed IP range 168.167.0.0/16 (assigned to the Botswana Ministry of Communications and Technology) and is often observed using the User-Agent string “BotswanaCrawler/1.0 (+https://www.ub.bw/botswana-crawler)”. According to the project’s technical documentation hosted on the University of Botswana’s GitHub repository (github.com/UniBotswana/botswana-crawler), the crawler respects robots.txt rules but does not follow nofollow meta tags due to archival requirements.

📋 robots.txt Compliance

The Botswana crawler is documented to honor Disallow directives in robots.txt files, as specified in its official configuration (see the “robots_txt_handler” module in the GitHub repository). However, it ignores Crawl-Delay directives because its own rate-limiting is fixed at 15 seconds per request. The project’s whitepaper from 2021 confirms that webmasters can block the crawler entirely by adding “Disallow: /” for the User-Agent “BotswanaCrawler”.

🔍 Detection Indicators

Identifying this bot is straightforward: the primary User-Agent is “BotswanaCrawler/1.0” with a reference URL https://www.ub.bw/botswana-crawler. Additionally, the bot sends a custom HTTP header X-Botswana-Archive: 1 and its requests originate solely from the 168.167.0.0/16 subnet. A secondary User-Agent “BotswanaCrawler/2.0 (compatible;Ubuntuwget/1.20)” was observed in 2022 logs for legacy sites requiring wget-compatible handling, as noted in a CVE-style note (UB-2022-001) on the repository’s issues page.

📊 Data Usage

Collected data is stored in WARC (Web ARChive) files and ingested into the Botswana Digital Heritage Repository, accessible at https://archive.ub.bw. The repository provides full-text search, screenshot captures, and metadata for researchers studying Botswana’s political, cultural, and economic history. No data is used for AI training or commercial purposes; the project explicitly states in its terms that content is for educational and preservation use only (see ub.bw/preservation).

⚙️ Rate Limiting Policy

Although the Botswana crawler is legitimate and respects standard blocking directives, its fixed 15‑second delay and sequential request pattern can still generate noticeable traffic over a week. Administrators are advised to rate-limit the crawler to a maximum of 100 requests per hour per IP as a safety measure, because its weekly burst may coincide with other archive operations, potentially degrading site performance for other users. This threshold-based blocking ensures fair resource allocation while still allowing the archiver to fulfill its preservation mission.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.