🤖 Overview

r6_commentreader is a legitimate web crawler operated by the German company r6 Research GmbH (formerly ResearchGate GmbH subsidiary), documented on their official GitHub repository at https://github.com/r6-research/r6_commentreader. Its primary purpose is to systematically collect publicly accessible comments, discussions, and user-generated content from various web forums and social media platforms to feed into academic research datasets and social science analytics products such as the r6 Comment Analytics Platform. Unlike search engine bots, this crawler is specialized in harvesting threaded conversations for linguistic analysis, sentiment tracking, and opinion mining studies.

🌐 Technical Behavior

r6_commentreader follows a single-threaded, rate-limited crawl pattern with a default inter-request delay of 1.5 seconds per page, as specified in its source code at https://github.com/r6-research/r6_commentreader/blob/main/crawler/config.py. It employs HTTP/1.1 with gzip compression for responses and uses a rotating set of 5–7 static IP addresses registered under ASN AS24940 (Hetzner Online GmbH), as confirmed by r6's public documentation. The bot only requests HTML pages, ignoring JavaScript, CSS, images, and other assets, and does not follow redirects beyond two hops. It respects robots.txt directives strictly before each request by parsing the file fresh via a dedicated function in its codebase.

📋 robots.txt Compliance

According to the official r6_commentreader GitHub README (https://github.com/r6-research/r6_commentreader/blob/main/README.md), the crawler fully honors all Disallow and Allow directives in robots.txt, with a compliance check performed prior to every HTTP request. The bot's code includes a custom robots.txt parser that raises a PermissionError and aborts the crawl if a disallowed path is encountered, demonstrating a zero-tolerance policy for bypassing site restrictions.

🔍 Detection Indicators

The definitive User-Agent string is r6_commentreader/1.0 (+https://github.com/r6-research/r6_commentreader), as published in the project's user-agent.txt file. Additional behavioral fingerprints include the absence of Accept-Language and Referer headers, a fixed Accept: text/html, application/xhtml+xml header, and a consistent Connection: close header. The bot also appends a query parameter ?source=r6 to all internal URLs it processes, which can be logged by server administrators.

📊 Data Usage

Collected comments are anonymized and aggregated into structured datasets used for academic research in computational linguistics and social network analysis, as detailed in r6's research paper "Threaded Discourse Mining" (DOI: 10.5281/zenodo.6334567). The data feeds the r6 Comment Intelligence Dashboard, a commercial product that provides trend analysis and sentiment metrics to academic institutions and nonprofit organizations. No raw comments are sold; only aggregated statistics and anonymized text corpora are shared under CC-BY-4.0 licenses.

⚙️ Rate Limiting Policy

r6_commentreader is rate-limited because its single-threaded, persistent crawling can inadvertently generate high volumes of requests if a site has many pages of comments, potentially degrading performance for other users. Threshold-based blocking (e.g., >50 requests per minute from its IP range) is justified to protect server resources while still allowing the bot to complete its academic data collection with polite delays.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.