🤖 Overview

The Typhoeus crawler is a legitimate, non‑malicious agent operated primarily by the Open Access Infrastructure for Research in Europe (OpenAIRE) and the Confederation of Open Access Repositories (COAR). It is designed to harvest metadata from Open Access repositories using the OAI‑PMH protocol, feeding structured records into aggregated discovery services such as CORE (core.ac.uk), BASE (base-search.net), and OpenAlex. The bot is named after the Typhoeus Ruby HTTP library used in its implementation, which enables efficient, parallel HTTP requests.

🌐 Technical Behavior

Typhoeus issues HTTP GET requests with a high degree of concurrency, typically sending 10–20 requests per second from IP ranges belonging to academic and research networks (e.g., 193.174.x.x for OpenAIRE). The crawler relies exclusively on the OAI‑PMH protocol (version 2.0), using ListRecords and ListIdentifiers verbs for selective harvesting. It supports Gzip compression to reduce bandwidth consumption and respects HTTP 429 Too Many Requests responses by backing off exponentially. The default crawl interval between successive requests is 1 second, configurable via the repository's OAI‑PMH and the Crawl‑delay directive. IPv4 and IPv6 are both supported.

📋 robots.txt Compliance

Typhoeus fully honors robots.txt directives. Official documentation from OpenAIRE states that the harvester checks robots.txt before every request and adheres to both Disallow and Crawl‑delay fields. It also respects X‑Robots‑Tag and noindex meta tags when harvesting OAI‑PMH endpoints that are also accessible via HTTP.

🔍 Detection Indicators

The primary User‑Agent string is Typhoeus (exact case). Additional identifying headers include From: [email protected] and User-Agent: Tyhpoeus/1.0 (Ruby; OAI‑PMH harvester). Behavioral fingerprints include sequential ListRecords requests, consistent low‑delay intervals, and the absence of JavaScript or image loading. The crawler never sends cookies or session headers.

📊 Data Usage

Collected metadata (titles, authors, abstracts, DOIs, ORCIDs) is ingested into large‑scale scholarly search indices. OpenAIRE uses the data to monitor European research output and funder compliance. CORE and BASE integrate the records to provide free discovery of millions of open‑access articles. The data is not used for AI model training or commercial analytics; it serves purely for academic indexing and bibliometric analysis.

⚙️ Rate Limiting Policy

Typhoeus is rate‑limited because its parallel harvesting can overwhelm underprovisioned OAI‑PMH endpoints, especially when multiple repositories are targeted simultaneously. A threshold‑based block (e.g., >60 requests per minute from a single IP) is recommended to protect server stability while still permitting the legitimate, non‑commercial harvesting that underpins open‑access infrastructure.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.