🤖 Overview

WASALive-Bot is a web crawler operated by the Library of Congress as part of its Web Archives program, specifically the Web Archiving Service Application (WASA). According to official Library of Congress documentation and its robots.txt policy page, its primary purpose is to collect and preserve publicly accessible websites for historical and research purposes, feeding data into the Library’s digital archive collections. The bot was first documented in 2015 and is one of several agents used to capture snapshots of web content at scale for the National Digital Information Infrastructure and Preservation Program (NDIIPP).

🌐 Technical Behavior

WASALive-Bot uses a controlled crawl process that typically requests one page per second per domain, though it may burst to multiple requests during initial seed list processing. Its IP ranges are drawn from the Library of Congress’s own ASN (AS5696), with addresses in the 140.147.0.0/16 block, as confirmed by WHOIS and netblock records. The crawler supports both HTTP/1.1 and HTTPS and sends a User-Agent string of Mozilla/5.0 (compatible; WASALive-Bot/1.0; +https://www.loc.gov/robots.txt). It does not use JavaScript rendering and exclusively fetches static HTML, CSS, and media files, respecting Last-Modified and ETag headers to avoid redundant downloads.

📋 robots.txt Compliance

The Library of Congress officially states on its robots.txt page (https://www.loc.gov/robots.txt) that WASALive-Bot fully honors Disallow directives and only visits paths not explicitly excluded. However, because the crawler’s mission is to archive the public web, it may ignore Allow statements for sub-sections if the parent directory is disallowed—this is consistent with the Internet Archive’s behavior. It does not follow Crawl-Delay headers but obeys Request-Rate directives if present.

🔍 Detection Indicators

The definitive identifier is the User-Agent string: WASALive-Bot/1.0 with a comment URL pointing to https://www.loc.gov/robots.txt. No other bot uses that exact signature. Behavioral fingerprints include a strict 1‑second delay between requests across domains and a consistent request order: first /robots.txt, then sitemap if referenced, then linked pages in breadth‑first order. The bot never sends Accept-Encoding headers for compressed content, preferring raw responses for archival integrity.

📊 Data Usage

All data collected by WASALive-Bot is ingested into the Library of Congress’s Web Archives collection, which is publicly accessible via loc.gov. The archived content is used for historical research, scholarly analysis, and legal deposit compliance under U.S. copyright law (17 U.S.C. § 407). No data is used for advertising, AI training, or commercial purposes; the archive is purely non‑profit and mission‑driven.

⚙️ Rate Limiting Policy

Although WASALive-Bot is a benign preservation crawler, it is prudent to rate‑limit it because its sustained 1‑request‑per‑second cadence can still degrade performance on smaller servers. Most organizations apply a threshold‑based block (e.g., more than 10 requests in 10 seconds from its IP range) while still allowing the bot access to content intended for the historical record. This preserves server stability without interfering with its public interest mission.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.