🤖 Overview

Infomine is a web crawler operated by Infomine Technologies, designed to index academic and research-oriented content for the Infomine search engine (www.infomine.com). First deployed in the early 2000s, it focuses on collecting publicly accessible scholarly articles, university resources, and government publications to populate its specialized database.

🌐 Technical Behavior

The crawler follows a breadth-first traversal pattern, requesting pages at intervals of 5–10 seconds, but can scale up to multiple concurrent requests when encountering large sites. It primarily uses HTTP/1.1 and respects standard request headers. Its IP addresses are drawn from a pool belonging to Infomine's own ASN (ASxxxxx) and occasionally from cloud providers, though the full range is not publicly documented. The bot identifies via the User-Agent string "Mozilla/5.0 (compatible; Infomine/1.0; +http://www.infomine.com/bot.html)" and does not spoof other agents.

📋 robots.txt Compliance

According to the official bot page (http://www.infomine.com/bot.html), Infomine fully honors robots.txt directives, including Disallow and Crawl-delay settings. Evidence from site administrators confirms that the bot halts crawling on restricted paths within one crawl cycle.

🔍 Detection Indicators

The primary indicator is the User-Agent "Infomine/1.0" with the trailing bot URL. Additionally, the bot includes a custom header "From: [email protected]" in some implementations. It does not exhibit unusual request patterns or hidden identifiers.

📊 Data Usage

Collected data feeds the Infomine search index, which specializes in academic and research materials. The index is used for public search and may also be used for internal analytics and trend monitoring in the scholarly domain. No evidence suggests use for AI model training.

⚙️ Rate Limiting Policy

Because the crawler can generate significant request volume on large sites, it is rate-limited to prevent server overload. A threshold of 100 requests per minute per IP is commonly applied by site operators, consistent with standard best practices for aggressive legitimate crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.