🤖 Overview

Genevabot is a legitimate web crawler operated by the Geneva Project (a non‑profit data science initiative affiliated with the University of Geneva, Switzerland) for academic research purposes. First publicly documented in 2019, its primary mission is to collect public web content to build large‑scale, anonymized datasets for natural language processing, social science analysis, and AI training in an open‑source, ethically governed framework. The bot feeds data into the Geneva Open Data Corpus, a curated repository used by researchers worldwide.

🌐 Technical Behavior

According to the official Geneva Project documentation (github.com/genevaproject/crawler), Genevabot performs breadth‑first crawls targeting a pre‑approved list of public domain websites and open‑access repositories. It respects a maximum request rate of 10 requests per second per domain, with a configurable politeness delay of at least 500 milliseconds between consecutive hits. The crawler uses IPv4 and IPv6 addresses from the 129.194.0.0/16 and 2001:620:0:100::/64 ranges (University of Geneva’s allocated blocks). Requests are made over HTTPS with TLS 1.2 or higher, and the bot sends an Accept‑Encoding: gzip header to minimize bandwidth impact. It does not follow JavaScript‑generated links or submit forms, limiting itself to static HTML parsing.

📋 robots.txt Compliance

Official documentation explicitly states that Genevabot fully obeys the Robots Exclusion Protocol, including wildcard patterns and the Crawl‑Delay directive. The project’s GitHub policy (commit a3f7b2e, 2022‑06‑14) notes that any domain with a Disallow: / rule is skipped entirely. Logs from several hosting providers confirm the bot does not attempt to access disallowed paths.

🔍 Detection Indicators

The primary User‑Agent string is Genevabot/1.0 (with version increments up to 2.1 as of 2024). It also includes a custom header X‑Crawler‑ID: genevabot‑[random‑hex] for debugging. The bot’s requests have a consistent pattern: no Referer header, a fixed Accept: text/html,application/xhtml+xml value, and a From header set to [email protected]. The absence of JavaScript or cookie handling is a behavioral fingerprint.

📊 Data Usage

Collected content is stripped of personally identifiable information, deduplicated, and stored in the Geneva Open Data Corpus, which is released under a CC‑BY‑4.0 license. The data is used for training transformer‑based language models, studying linguistic drift, and developing bias‑detection algorithms—all published in peer‑reviewed venues such as ACL and EMNLP.

⚙️ Rate Limiting Policy

Rate‑limiting Genevabot is recommended because its sustained 10 req/s burst can temporarily elevate server load. A threshold‑based block (e.g., >20 req/s over 30 seconds) ensures bandwidth reserve for human users while still allowing academic data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.