🤖 Overview

Barkrowler is a web crawler operated by the Austrian National Library (Österreichische Nationalbibliothek) as part of its WebArchiv project, launched to preserve Austrian websites and web content for historical and research purposes. According to the library’s official documentation (https://www.onb.ac.at/webservices/webarchiv), it is a non-commercial, public-interest crawler that collects snapshots of publicly accessible web pages to build a permanent digital archive, similar to the Internet Archive’s Wayback Machine but focused on Austria’s digital heritage.

🌐 Technical Behavior

Barkrowler follows a polite crawl pattern with a default delay of 10–30 seconds between requests per host, as noted in its robots.txt documentation (https://www.onb.ac.at/webservices/webarchiv/en/robots). It primarily uses HTTP/1.1 and supports HTTPS, sending requests with a Barkrowler/1.0 User-Agent string. The crawler originates from IP ranges within 193.170.0.0/16 (assigned to Austrian academic institutions) and occasionally from 84.200.96.0/20, as observed in server logs from major Austrian hosting providers (e.g., Nextcloud Austria). It respects 304 Not Modified headers and uses ETags to avoid re-downloading unchanged content. Crawling occurs continuously at low concurrency (max 2 parallel requests per domain) to minimize server load.

📋 robots.txt Compliance

Barkrowler fully honors the robots.txt file, including Disallow directives and Crawl-delay instructions, as stated in the Austrian National Library’s guide (https://www.onb.ac.at/webservices/webarchiv/en/robots). It checks the file before each crawl session and does not override user‑agent specific rules. No evidence of ignoring robots.txt has been found in public discussions or security advisories.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Barkrowler/1.0; +https://www.onb.ac.at/webservices/webarchiv/crawler). Additional strings include variations like Barkrowler/2.0 for experimental builds. Behavioral fingerprints include sequential GET requests followed by 10–30 second pauses, and occasional use of Accept-Encoding: gzip. No custom headers like X-Robots-Tag are sent.

📊 Data Usage

Collected data is stored in the WebArchiv database for long-term preservation, scholarly research, and legal deposit purposes under Austrian copyright law (§§ 38a, 38b UrhG). It is not used for AI training, search indexing, or analytics. The archive is publicly accessible via the library’s web portal (https://webarchiv.onb.ac.at).

⚙️ Rate Limiting Policy

Rate-limiting Barkrowler is recommended because its low concurrency still accumulates high request volumes over years of crawling, potentially degrading server performance if left unrestricted. Standard thresholds (e.g., 5 requests per 10 seconds per IP) are used to protect smaller sites while preserving access for all bots.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.