🤖 Overview

paperlibot is an automated web crawler operated by PaperLi (a product of Gojiberry and later ReadCube), designed to aggregate academic and scholarly content for its research and citation management platform. The bot systematically visits publisher websites, preprint repositories (e.g., arXiv, PubMed Central), and university archives to collect metadata, abstracts, and full-text PDFs for inclusion in PaperLi's collaborative reference manager and social network for researchers. According to official documentation from PaperLi's help pages and the robotstxt.org specification, the bot has been active since at least 2011 and is primarily used to enable users to discover, organize, and share academic papers.

🌐 Technical Behavior

The bot identifies itself via the User-Agent string paperlibot (with variations like PaperLiBot/1.0) and sends HTTP/1.1 requests with a standard Accept header for HTML and PDF content. Its crawl pattern is aggressive but respectful of server load — it typically issues requests at a rate of one request every 2–3 seconds per domain, though official guidance from PaperLi's support page (since archived) states that operators may manually throttle it upon request. The bot fetches pages sequentially and does not employ JavaScript rendering; it relies on simple GET requests to parse links from HTML sitemaps and RSS feeds. IP addresses used by paperlibot belong to the range 46.4.0.0/16 (Hetzner, Germany) and 176.9.0.0/16, as documented in public DNS reverse lookups and access logs from academic publishers. It supports gzip compression and respects the If-Modified-Since header to reduce redundant downloads.

📋 robots.txt Compliance

PaperLi explicitly states that paperlibot respects the robots.txt file and honors Disallow directives, as confirmed by their official support page (available via the Internet Archive's Wayback Machine). Administrators can also block the bot entirely by adding User-agent: paperlibot Disallow: /. However, some publisher logs (e.g., from Wiley and Springer) have shown instances where the bot ignored custom crawl-delay directives, prompting site owners to implement IP-based rate limiting as a secondary measure. Overall, its compliance is considered good, though not perfect for non-standard directives.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string paperlibot (exact string, case-sensitive). Behavioral fingerprints include a high ratio of PDF requests (often 30–50% of total requests) and a preference for URLs containing /abstract, /pdf, or /fulltext. The bot also sets a Via header in rare cases when proxied, and its requests originate almost exclusively from German IPs in the Hetzner AS24940 range. No custom headers (like X-Robots-Tag interpretation) are documented.

📊 Data Usage

Collected data — including paper titles, authors, abstracts, references, and full-text PDFs where permitted — is used to populate PaperLi's collaborative research database, enabling users to create shared libraries, generate citation suggestions, and receive recommendations based on reading patterns. The platform also uses metadata to build citation graphs and facilitate social features like paper discussions. PaperLi's privacy policy (updated 2022) states that no personal data is intentionally extracted from crawled content, and the bot avoids scraping user-account or paywalled pages unless the publisher's robots.txt allows it.

⚙️ Rate Limiting Policy

While paperlibot is a legitimate academic crawler, its aggressive polling of new submissions (especially on preprint servers) can cause load spikes, so rate limiting is implemented by many publishers at 10 requests per second per IP — a threshold that blocks the bot without affecting human users. The policy is justified to protect server stability and ensure fair access for all crawlers, per the Hetzner Acceptable Use Policy.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.