🤖 Overview

Widow is a web crawler operated by the Widow Archiving Project, an academic initiative at the University of Amsterdam, first documented in a 2022 research paper. Its primary purpose is to collect publicly accessible web content for digital preservation, historical analysis, and training of archival AI models as part of the European Web Archive initiative. The bot is designed for non-commercial, scholarly use and feeds data into the Widow Digital Repository, a public dataset of timestamped web snapshots.

🌐 Technical Behavior

Widow crawls by sending HTTP GET requests with a variable frequency ranging from 1 request per 5 seconds to 1 request per 30 seconds, depending on the site’s responsiveness. It operates from a fixed IP range registered to the University of Amsterdam (145.100.0.0/16) and uses IPv4 and IPv6 addresses. The crawler obeys a polite crawl rate based on the crawl-delay directive in robots.txt but also implements exponential backoff when encountering server errors. Widow uses a custom asynchronous Python framework (disclosed on its GitHub repository at github.com/widow-archive/crawler) and supports HTTP/1.1 and HTTP/2 protocols. It does not execute JavaScript or submit forms, focusing solely on static HTML and linked resources (CSS, images, PDFs) with allowed content types declared in its Accept header.

📋 robots.txt Compliance

According to the Widow Archiving Project’s official documentation (widowarchive.org/robots), the crawler fully honors Disallow directives and respects Crawl-Delay entries. It also checks the X-Robots-Tag header for noindex and nofollow instructions. The project’s policy mandates that all archived pages be publicly accessible, so they exclude any content explicitly disallowed. Evidence from the project’s transparency reports shows they have never been reported for violating robots.txt rules.

🔍 Detection Indicators

The primary User-Agent string is Widow/1.0 (+https://widowarchive.org/bot), though variants such as Widow-Crawler/1.2 have been observed. Behavioral fingerprints include a sequential request pattern without randomization, a distinctive Accept header (text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8), and a custom HTTP header X-Widow-Archive: true. The bot also identifies via the From header with the email address [email protected].

📊 Data Usage

Collected data is used solely for academic research and digital preservation, with snapshots stored in the Widow Digital Repository under a Creative Commons Zero license. The dataset is used to train archival AI models for automated metadata extraction, duplicate detection, and temporal link analysis. No commercial reuse or third-party API access is provided; data can be downloaded from the project’s public S3 bucket as WARC files.

⚙️ Rate Limiting Policy

Widow is rate-limited because its sustained sequential crawling can still generate a high volume of requests over time, potentially impacting site performance for shared hosting environments. A threshold-based block is applied if the crawler exceeds 30 requests per minute, with a 24-hour cooling period, aligning with the project’s own polite crawl standards to prevent server overload while preserving its legitimate archiving mission.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.