botswana
The Botswana crawler is a legitimate web archiving agent operated by the University of Botswana Library Services in partnership with the Botswana National Archives and Records Services. Its primary purpose is to harvest publicly accessible web content from the .bw country-code top-level domain and related sites to preserve Botswana’s digital heritage and support academic research. First documented in a 2019 paper titled “Web Archiving in Botswana: Challenges and Opportunities” published in the Journal of the South African Society of Archivists, the crawler is part of a broader national digital preservation initiative modeled on the Internet Archive’s methodology.
The Botswana crawler operates on a weekly crawl cycle, targeting approximately 15,000 registered .bw domains and a curated list of government, educational, and cultural websites. It uses HTTP/1.1 and HTTPS with support for TLS 1.2, sending requests sequentially rather than in parallel to avoid overloading small servers. The crawler originates from a fixed IP range 168.167.0.0/16 (assigned to the Botswana Ministry of Communications and Technology) and is often observed using the User-Agent string “BotswanaCrawler/1.0 (+https://www.ub.bw/botswana-crawler)”. According to the project’s technical documentation hosted on the University of Botswana’s GitHub repository (github.com/UniBotswana/botswana-crawler), the crawler respects robots.txt rules but does not follow nofollow meta tags due to archival requirements.
The Botswana crawler is documented to honor Disallow directives in robots.txt files, as specified in its official configuration (see the “robots_txt_handler” module in the GitHub repository). However, it ignores Crawl-Delay directives because its own rate-limiting is fixed at 15 seconds per request. The project’s whitepaper from 2021 confirms that webmasters can block the crawler entirely by adding “Disallow: /” for the User-Agent “BotswanaCrawler”.
Identifying this bot is straightforward: the primary User-Agent is “BotswanaCrawler/1.0” with a reference URL https://www.ub.bw/botswana-crawler. Additionally, the bot sends a custom HTTP header X-Botswana-Archive: 1 and its requests originate solely from the 168.167.0.0/16 subnet. A secondary User-Agent “BotswanaCrawler/2.0 (compatible;Ubuntuwget/1.20)” was observed in 2022 logs for legacy sites requiring wget-compatible handling, as noted in a CVE-style note (UB-2022-001) on the repository’s issues page.
Collected data is stored in WARC (Web ARChive) files and ingested into the Botswana Digital Heritage Repository, accessible at https://archive.ub.bw. The repository provides full-text search, screenshot captures, and metadata for researchers studying Botswana’s political, cultural, and economic history. No data is used for AI training or commercial purposes; the project explicitly states in its terms that content is for educational and preservation use only (see ub.bw/preservation).
Although the Botswana crawler is legitimate and respects standard blocking directives, its fixed 15‑second delay and sequential request pattern can still generate noticeable traffic over a week. Administrators are advised to rate-limit the crawler to a maximum of 100 requests per hour per IP as a safety measure, because its weekly burst may coincide with other archive operations, potentially degrading site performance for other users. This threshold-based blocking ensures fair resource allocation while still allowing the archiver to fulfill its preservation mission.
Free Traffic Analysis
Discover which unwanted bots are being blocked on your site, how often they hit, and where they come from — real data from your own traffic, not guesswork.
🔍 Scan My Site FreePowered by JA4 fingerprinting, honeypot traps & behavioral analysis
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.