🤖 Overview

MendeleyBot is a web crawler operated by Mendeley Ltd., a subsidiary of Elsevier, as part of the Mendeley reference management and academic social network platform. First documented around 2012, its primary purpose is to collect publicly accessible metadata, abstracts, and full-text references from scholarly articles, conference proceedings, and institutional repositories to populate Mendeley’s citation database and enable features like automatic paper discovery, recommendation, and cross-referencing for researchers. The bot is a legitimate, non-malicious agent that supports academic indexing; it is not associated with any threat actors or attack tools.

🌐 Technical Behavior

MendeleyBot performs GET requests for HTML pages and PDF files, typically crawling at a moderate rate of a few requests per second to avoid overwhelming servers. It follows standard HTTP/1.1 protocols and uses the User-Agent string "MendeleyBot/1.0 (http://www.mendeley.com)" or variations like "MendeleyBot/2.0" for newer versions. The crawler originates from a range of IP addresses owned by Elsevier’s cloud infrastructure, including those assigned to AWS and Azure data centers, with no fixed IP range published; however, system administrators can reverse-lookup the hostnames (e.g., *.mendeley.com or *.elsevier.com) for verification. It primarily targets URLs containing "/articles/", "/doi/", and repository subdomains, and it often respects HTTP status codes such as 429 (Too Many Requests) by backing off, based on observed behavior in web server logs.

📋 robots.txt Compliance

According to Mendeley’s published crawl policy and industry standards, MendeleyBot respects the robots.txt file’s Disallow directives. Official documentation from Mendeley’s help center (currently hosted at https://support.mendeley.com/) states that the bot will honor explicit exclusions. Evidence from webmaster forums and robotstxt.org records confirms that MendeleyBot does not crawl paths listed in Disallow rules, though it may ignore Crawl-Delay directives if not set; administrators are advised to use explicit Disallow to block unwanted sections.

🔍 Detection Indicators

The primary detection indicator is the User-Agent header: "MendeleyBot/1.0 (compatible; Mendeley; http://www.mendeley.com)" or "MendeleyBot/2.0". Additionally, the crawler often includes a From header set to "[email protected]" and a Accept header that prioritizes text/html and application/pdf. Behavioral fingerprints include a consistent request interval of 2–5 seconds, no JavaScript execution, and a preference for DOI-based URLs. No known CVE entries are associated with MendeleyBot as it is not a threat; it is purely a benign indexing agent.

📊 Data Usage

The data collected by MendeleyBot is exclusively used for academic indexing and citation management within the Mendeley platform. Metadata—such as article titles, authors, abstracts, publication dates, and references—is stored in Mendeley’s cloud database to enable users to organize research libraries, generate bibliographies, and discover related papers through recommendation algorithms. No full-text content is stored or used for commercial AI training; the bot only indexes publicly accessible pages and respects copyright restrictions, as stated in Mendeley’s terms of service.

⚙️ Rate Limiting Policy

MendeleyBot is rate-limited because its crawling, while legitimate and well-behaved, can generate significant load on smaller academic repositories and institutional servers, especially when many papers are indexed simultaneously. A typical rate-limit threshold of 10 requests per second per IP is recommended by web administrators to maintain fair access; if exceeded, the bot will pause and retry later, as per its built-in exponential backoff algorithm documented in Mendeley’s developer notes.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.