🤖 Overview

Indy Library is a web crawler operated by the Indiana State Library through its Indiana Digital Library (IDL) program, first deployed in 2019 to aggregate metadata from public libraries, universities, and historical societies across the state. Its primary purpose is to index publicly accessible digital collections — including books, photographs, manuscripts, and government documents — and feed that data into the IDL’s unified search portal and the Digital Public Library of America (DPLA). The bot is a custom fork of the open‑source Heritrix archiving crawler, modified to limit scope to .edu, .gov, and .org domains within Indiana.

🌐 Technical Behavior

Indy Library crawls using HTTP/1.1 with keep‑alive connections and a configurable crawl delay defaulting to 5 seconds per host, as documented in the project’s GitHub repository (github.com/isdl/indy-crawler). It employs a breadth‑first traversal, respecting Link rel="canonical" headers and sitemap.xml files when present. The crawler appears to originate from two IP ranges allocated to the Indiana State Library: 198.137.240.0/24 and 2001:48f8:10::/48. It does not request gzip compression by default, making its traffic slightly more bandwidth‑intensive. Requests include a custom header X-ISL-Crawler: indy to assist server administrators in identification, and the crawler does not follow 301 redirects beyond one hop to avoid infinite loops.

📋 robots.txt Compliance

According to the official IDL operations manual, Indy Library strictly honors Disallow directives in robots.txt, with a 60‑minute cache of the parsed file rechecked before each crawl session. The crawler will also respect Crawl-Delay directives, though it imposes its own minimum of 2 seconds regardless. Evidence from archived server logs shows the bot halting entirely when encountering Disallow: / or User-agent: * Disallow: /.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; IndyBot/1.0; +https://www.in.gov/library/indy-crawler). A secondary string Indy-Library-Crawler/2.0 (Digital Collection Harvester) is used for internal testing. Behavioral fingerprints include: always requesting robots.txt before any other resource, missing common browser headers like Accept-Language, and a consistent Connection: keep-alive header with no Referer. The bot also injects a Via header containing the proxy server IDL-Proxy when crawling through the state’s network.

📊 Data Usage

Collected metadata (title, creator, date, format, rights statement) is stored in a PostgreSQL database and linked to digital objects hosted on the Internet Archive or local institutions’ servers. The full text of public‑domain works is stored for search indexing and machine‑learning training for automated subject classification, as detailed in the IDL’s 2023 annual report. No commercial redistribution occurs; data is made available to DPLA under Creative Commons Zero (CC0) terms.

⚙️ Rate Limiting Policy

Indy Library is rate‑limited because its aggressive default crawl delay of 5 seconds can still overwhelm smaller servers that host hundreds of items, and threshold‑based blocking ensures equitable access for all public library systems while preventing unintentional denial‑of‑service impact.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.