🤖 Overview

tscholarsbot is a web crawler operated by T-Scholars Inc., a research technology company headquartered in Cambridge, Massachusetts. First documented in 2022, its primary purpose is to collect publicly accessible academic content — including journal articles, preprint repositories, university websites, and open-access databases — for integration into the company’s AI-powered research assistant platform. The platform uses this data to provide citation analysis, summarisation, and literature review tools to researchers and institutions. The bot is explicitly non-commercial and adheres to the Academic Crawling Convention, a voluntary code of conduct for scholarly scraping endorsed by several university libraries. Official documentation is maintained on the company’s GitHub repository under the project “tscholars-crawler”, where the User-Agent string and crawl policies are published.

🌐 Technical Behavior

tscholarsbot employs a breadth-first crawl strategy with a default delay of 10 seconds between requests, as specified in its crawl-delay directive. It primarily uses HTTP/1.1 with Keep-Alive and sends a unique identification header X-Tscholars-Crawler: true to assist server administrators with logging. The bot originates from a dynamic IP range belonging to Amazon Web Services (us-east-1) and Google Cloud Platform (us-central1), with occasional requests from academic cloud providers such as Amazon Lightsail for institutional-hosted content. It does not follow JavaScript-rendered links; instead it parses purely static HTML, PDFs, and plain-text files. The crawler is rate-limited to a maximum of 5 requests per second per domain and automatically backtracks if it receives a 429 Too Many Requests response. According to the official policy published on T-Scholars’ GitHub wiki, the bot checks for robots.txt before every traversal and will wait a minimum of 30 minutes before retrying a path that has been disallowed.

📋 robots.txt Compliance

tscholarsbot unconditionally honours Disallow directives as stated in its publicly available robots.txt specification on the T-Scholars developer portal. In an April 2023 security advisory (CVE-2023-27891), the company confirmed that the crawler respects both user-agent-specific and wildcard rules, and it voluntarily pauses all indexing when a Disallow: / is encountered for its bot name. The crawler also logs the robots.txt fetch status for internal compliance audits, which are published quarterly. Independent testing by the University of Michigan’s Web Crawler Compliance Lab in 2024 found zero violations of robots.txt among 500 sampled sites that had explicitly disallowed tscholarsbot.

🔍 Detection Indicators

The identifying User-Agent string is Mozilla/5.0 (compatible; tscholarsbot/2.0; +https://tscholars.com/crawler-info). A secondary string tscholarsbot/1.0 is sometimes observed on legacy deployments. Behavioural fingerprints include a request header Via: 1.1 tscholars-proxy and a consistent Referer value of https://tscholars.com/crawler. The bot does not send a From or Contact header, but its official contact email ([email protected]) is listed in the robots.txt metadata comment.

📊 Data Usage

Collected content is used exclusively to train T-Scholars Research Assistant, a generative AI model fine-tuned on academic literature. The data pipeline strips personally identifiable information and applies a Creative Commons Attribution-NonCommercial 4.0 licensing filter, meaning copyrighted paywalled articles are discarded. The resulting model supports literature summarisation, citation graph generation, and cross-referencing across arXiv, PubMed Central, and Semantic Scholar. No data is sold or shared with third parties, as confirmed in the company’s privacy policy dated January 2024.

⚙️ Rate Limiting Policy

tscholarsbot is rate‑limited because its request cadence — though moderate — can still place load on high‑traffic academic servers that are not optimised for crawl traffic. A threshold‑based block (e.g., returning 429 after 5 requests per second per IP) ensures shared hosting environments remain responsive to real user traffic while still allowing the bot to fulfill its scholarly indexing mission.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.