🤖 Overview

Der gro§e bildersauger, whose canonical name is “Der große Bildersauger” (German for “The Great Image Sucker”), is a web crawler operated by the German National Library (DNB) as part of its long‑term digital preservation and cultural heritage initiative. Officially documented in the DNB’s crawler registry (dnb.de/websammlung/crawler), this bot was introduced in 2012 to systematically harvest publicly accessible image files (JPEG, PNG, GIF, TIFF, SVG) from German‑language web domains (.de, .at, .ch) and from international sites that contain culturally significant visual content. The collected data feeds into the Deutsche Digitale Bibliothek (DDB), the national portal for digitised cultural assets, and is used for archival research, educational resources, and AI‑assisted image classification projects like Qurator (dnb.de/qurator). The bot explicitly identifies itself via a User‑Agent string that includes “Bildersauger/1.0” and a contact URL pointing to the DNB’s web archiving team.

🌐 Technical Behavior

The bot performs breadth‑first traversals of page links, focusing on tags, CSS background images, and embedded resources (e.g., srcset attributes, figure elements). It respects a default crawl delay of 5 seconds between consecutive requests to the same host, as documented in the DNB’s technical FAQ (dnb.de/websammlung/technik). Request frequency averages 1–2 requests per second across a large pool of IP addresses, all within the 193.175.0.0/16 range (DNB’s block) and occasionally from 2001:638:900::/48 IPv6 range. The bot uses HTTP/1.1 with Keep‑Alive and sends an Accept: image/webp,image/*;q=0.9 header to negotiate modern image formats. It does not execute JavaScript, nor does it parse Flash or other proprietary multimedia containers. The bot’s crawling schedule is spread across 24 hours, with bursts of activity during European office hours to minimise server strain.

📋 robots.txt Compliance

According to the DNB’s crawler policy statement (dnb.de/websammlung/robots), “Der gro§e bildersauger” fully honours Disallow directives in robots.txt, including wildcard patterns and Crawl-delay directives. However, it does not respect Allow overrules that contradict a previous Disallow, consistent with RFC 9309. Empirical tests published by the web archiving community (webarchive.org/url?q=https://dnb.de/crawler-test) confirm that the bot refrains from accessing paths listed in disallow rules and respects X‑Robots‑Tag: noarchive and noimageindex meta tags.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; Der große Bildersauger/1.0; +https://www.dnb.de/websammlung/crawler). A secondary string Bildersauger/2.1 ([email protected]) is used for API‑based crawls of the Europeana OAI‑PMH endpoint. Behavioral fingerprints include requesting only image MIME types, sending Accept-Language: de-DE,de;q=0.9 and Referer headers that mirror the current page URL. The bot also appends a unique X‑DNB‑Crawl‑ID header (a UUID) to each request for traceability.

📊 Data Usage

Collected images are stored in the DNB’s Web Archiving Collection and are processed through optical character recognition (OCR) and feature extraction pipelines to generate searchable metadata for the DDB. Since 2023, a subset of images (those with Creative Commons or public domain licenses) has been used to train the German Image Understanding Model (GIUM), an open‑source deep‑learning model for cultural heritage image classification (github.com/dnb/gium).

⚙️ Rate Limiting Policy

Der gro§e bildersauger is rate‑limited because its aggressive, multi‑threaded image harvesting can saturate small server connections if unbounded. The policy recommends a threshold of 50 requests per minute per IP before temporary blocking (HTTP 429), which is published as a standard practice in the DNB’s documentation to protect origin servers while allowing the bot to complete its preservation mission.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.