🤖 Overview

omniweb is an automated web crawler operated by Semantic Scholar, a research platform developed by the Allen Institute for AI (AI2). Its primary purpose is to discover and index academic papers, conference proceedings, and scholarly content available on the public web, feeding data into the Semantic Scholar search engine and knowledge graph. First documented in 2015, omniweb helps maintain Semantic Scholar’s comprehensive corpus of over 200 million research publications. According to official documentation on semanticscholar.org, the crawler is designed specifically for academic content and respects standard web crawling conventions. It is not associated with any threat actors or malicious activity.

🌐 Technical Behavior

omniweb performs breadth-first crawls focused on domains hosting PDFs, HTML abstracts, and metadata from journals, preprints servers (e.g., arXiv, bioRxiv), and university repositories. It requests robots.txt before each crawl session and obeys Crawl-Delay directives. The bot’s request frequency is moderate, typically issuing one request every few seconds per host, but it may burst when encountering large PDF collections. IP ranges are allocated from the AWS EC2 and Google Cloud Platform address pools, as reported in published crawler documentation and observed in web server logs. The user-agent string omniweb (crawler; semanticscholar.org; contact: [email protected]) is used in the HTTP User-Agent header. The bot retrieves resources over HTTPS and employs ETag and If-Modified-Since headers to reduce redundant downloads. It also parses structured metadata from HTML meta tags, Dublin Core, and schema.org JSON-LD.

📋 robots.txt Compliance

omniweb is fully compliant with robots.txt directives, according to the official Semantic Scholar crawler policy published at semanticscholar.org/crawler. It honors both Disallow and Crawl-Delay directives. The crawler’s documentation explicitly states that it will not crawl any path disallowed in robots.txt and that it respects HTTP 503 retry-after headers. Webmasters can block omniweb entirely by adding User-agent: omniweb followed by Disallow: /.

🔍 Detection Indicators

The primary indicator is the User-Agent string: omniweb (crawler; semanticscholar.org; contact: [email protected]). No other user-agent variants have been documented. The bot also includes a From header with the same email address. Behavioral fingerprints include sequential request patterns with inter-request intervals consistent with a polite crawler, and a preference for application/pdf and text/html content types. Web logs also show the bot sending requests from IP ranges listed in the Semantic Scholar crawler IP list (published on their website). It never sends fake user-agents or modifies headers to disguise itself.

📊 Data Usage

Collected data is used solely to enhance the Semantic Scholar academic search engine and AI-powered recommendation system. The platform uses crawled papers to extract citations, figures, and author metadata for training natural language processing models that improve paper relevance ranking and summarization. Data is also used to build the Open Research Corpus, a public dataset of scholarly articles. No commercial third-party or training of general-purpose LLMs is involved. The crawler does not index personal data, user accounts, or subscription-only content without explicit access.

⚙️ Rate Limiting Policy

omniweb is rate-limited by server administrators because its broad academic crawl can generate significant traffic when discovering new repositories. Threshold-based blocking (e.g., >5 requests per second) is a reasonable policy to protect server performance while still allowing the useful indexing of publicly available scholarly materials.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.