koninklijke
The Koninklijke bot is a web crawler operated by the Koninklijke Bibliotheek (KB), the National Library of the Netherlands, as part of the Dutch Web Archive initiative (Nederlands Webarchief). Its primary purpose is to systematically harvest publicly accessible Dutch web content for long-term digital preservation, ensuring that cultural heritage is preserved for future generations of researchers and historians. The crawler feeds data into the KB’s archival storage systems, which are accessible on‑site in the Library’s reading rooms.
The crawler is built on the Heritrix open‑source web archiving framework developed by the Internet Archive, specifically using version 3.x with custom modules for handling Dutch‑language content and date‑based scope rules. According to the official KB documentation available at https://www.kb.nl/en/organisation/kb-crawler, the bot makes one request per second per host by default, but dynamically adjusts based on the Crawl-Delay directive in robots.txt. It uses the HTTP/1.1 protocol and follows redirects while archiving only the final destination. The IP ranges are allocated from the SURFnet academic network, specifically within the 145.100.0.0/16 and 84.24.128.0/18 blocks, with reverse DNS hostnames of the form kb-crawler-*.kb.nl. The crawler runs continuously, revisiting archived sites on a schedule that varies from monthly for high‑priority domains to annually for less frequently updated content.
The Koninklijke crawler fully respects the Robots Exclusion Protocol, including both Disallow and Allow directives. The official KB policy states that the crawler will not index pages blocked by robots.txt, and it also honours the Crawl-Delay instruction to throttle its request rate as specified by site owners. Evidence of this compliance can be found in the KB’s public description published at the same URL referenced above.
The primary User‑Agent string is Mozilla/5.0 (compatible; KB-Crawler/1.0; +https://www.kb.nl/en/organisation/kb-crawler). Additional identification includes the From header set to [email protected] (configurable) and the Accept-Language header defaulting to nl,en;q=0.8. Behavioral fingerprints include consistent HTTP/1.1 requests without compression headers and a predictable timing pattern of exactly one request per second unless overridden by robots.txt. The crawler does not obfuscate its identity and does not use dynamic IP rotation.
The collected data is stored in the Dutch Web Archive for long‑term preservation and is made available exclusively on‑site at the KB reading rooms for research purposes. Data is not used for AI model training, advertising, or commercial analytics; its sole purpose is cultural heritage archiving. The KB also provides a public interface (webarchief.kb.nl) for browsing the archived content within the legal framework of Dutch copyright laws.
Rate limiting is recommended because the crawler’s fixed one‑request‑per‑second pace can become aggressive when archiving very large sites (e.g., those with millions of URLs), and threshold‑based blocking ensures that the crawler does not degrade performance for human visitors while still allowing it to complete its archival mission within a reasonable timeframe. The KB requests site operators to use Crawl-Delay rather than blocking the crawler entirely.
Similar Threats
Free Bot Analysis
Find out exactly how much of your traffic is automated — and which bots are draining your bandwidth and skewing your analytics.
Run Free Bot Scan →No credit card required · Results in minutes
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.