🤖 Overview

Webis is an academic web crawler operated by the Webis research group at Bauhaus University Weimar in Germany, originally developed for building large-scale web corpora such as the ClueWeb09 and ClueWeb12 datasets used in information retrieval and natural language processing research. The crawler also supports the TIR (Text and Information Retrieval) Lab’s ongoing studies on web spam detection, adversarial crawling, and archival analysis, as documented on the official webis.de website and related publications.

🌐 Technical Behavior

The Webis crawler implements a focused, breadth-first crawl strategy with dynamic politeness delays that adapt based on server response times, typically maintaining a request rate of 1‑10 requests per second per domain. It originates from a pool of IPv4 addresses within the 141.54.x.x and 192.168.x.x ranges (university network), though recent deployments may use cloud-based proxies for scalability. The crawler follows HTTP/1.1 and HTTPS protocols, sending a "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" header and a custom User-Agent string. It downloads only publicly accessible pages and does not submit forms or access authenticated resources, as per its design described in the Webis technical reports available on the group’s GitHub repository.

📋 robots.txt Compliance

The official Webis crawler fully honors robots.txt directives, including Disallow rules and Crawl-delay hints, as stated in the crawl configuration files released in their open‑source tool Scrapy‑based crawler on GitHub (github.com/webis-de/webis-crawler). The group explicitly documents that their crawler pauses for the specified delay before fetching subsequent pages from a host, and they encourage website owners to use robots.txt to control access. Evidence from academic papers on the ClueWeb12 corpus confirms that the crawler respects these rules meticulously.

🔍 Detection Indicators

Primary detection is via the User-Agent string: "Mozilla/5.0 (compatible; webis-crawler/1.0; +https://webis.de/crawler.html)" or variations like "WebisCrawler/1.0". Additional HTTP headers include "X‑Robots‑Tag: noindex" (if present) and a custom "From: [email protected]" email header for contact. The crawler often appends a query parameter ?source=webis to URLs for traceability. Reverse DNS lookups reveal hostnames under uni-weimar.de or webis.de domains.

📊 Data Usage

Collected data is used exclusively for academic research, including training and evaluation of ranking algorithms, spam filtering models, and web archiving systems. The ClueWeb09 and ClueWeb12 datasets, totaling over 1 billion English web pages, are publicly distributed for non‑commercial research via the Lemur Project and the CAR (Center for Automated Research) at the University of Massachusetts Amherst. The Webis group also uses the crawler for longitudinal studies of web evolution and adversarial attacks on machine learning models.

⚙️ Rate Limiting Policy

While Webis is a legitimate academic crawler, its moderate request frequency can still place load on under‑provisioned servers, justifying a rate‑limiting threshold of 10 requests per second per IP to prevent resource exhaustion. Administrators may block or throttle if the crawler exceeds polite delays due to misconfiguration or aggressive parallelization in research experiments.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.