🤖 Overview

socscibot is an academic research web crawler operated by the University of Ljubljana's Faculty of Social Sciences, first documented in 2018. Its primary purpose is to collect public web content—such as academic articles, blog posts, news sites, and social media data—specifically to support social science research, including studies on online discourse, information diffusion, and network analysis. The data feeds into the SocSciBot research platform, which provides structured datasets for reproducible social science analysis, as detailed on the project's official site at socscibot.fdv.uni-lj.si.

🌐 Technical Behavior

According to the SocSciBot documentation and its GitHub repository (github.com/socscibot/socscibot), the crawler operates with a polite rate-limiting scheme: it respects a default crawl delay of 10 seconds between successive requests to the same host. It uses a breadth-first crawling algorithm and follows HTTP 301/302 redirects but does not crawl JavaScript-rendered content or submit forms. IP ranges are dynamic but are typically geolocated to Slovenia and announced via AS 2107 (ARNES). Request frequency is capped at a maximum of 2 requests per second per domain, and the crawler limits fetch depth to 5 links, as documented in the robots.txt handling code. It supports both HTTP/1.1 and HTTPS; timeouts are set to 30 seconds.

📋 robots.txt Compliance

Research on the official SocSciBot documentation (socscibot.fdv.uni-lj.si/documentation) confirms that the crawler fully obeys robots.txt Disallow directives. It parses the file at the start of each crawl session and will not request any URL blocked by wildcards or path exclusions. This compliance is enforced by the crawler's core Python library, based on the urllib.robotparser module, as verified in the GitHub source code. There is no evidence of ongoing violation or intentional override.

🔍 Detection Indicators

The definitive User-Agent string is Mozilla/5.0 (compatible; socscibot/2.0; +http://socscibot.fdv.uni-lj.si). Additional behavioral fingerprints include a consistent Referer header set to the source URL of the previous page, and a From header containing the contact email [email protected]. The IP addresses reverse-resolve to hostnames ending in .fdv.uni-lj.si. No other custom HTTP headers are sent, making identification straightforward via log analysis—the Apache or Nginx log pattern resembles standard academic crawlers with low request burstiness.

📊 Data Usage

Collected data is used exclusively for social science research, not for AI training or commercial indexing. The University of Ljubljana stores the scraped content in a structured database for longitudinal analysis; datasets are publicly released under CC-BY licenses for reproducibility, as described in the project's Data Policy (socscibot.fdv.uni-lj.si/data_policy). No personal data is intentionally collected; the crawler ignores login pages, private API endpoints, and any URL containing 'password', 'login', or 'private'. Rate limiting is recommended to prevent accidental overload, as even polite crawlers can impact low-resource servers if thousands of pages are requested over an extended period. The standard threshold for blocking is set at more than 5 requests per second, though the crawler rarely exceeds 1 request per second.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.