🤖 Overview

Webleacher is an open-source web crawler operated by the Web Leacher Project, a community-driven initiative aimed at collecting publicly available web content for training large language models and AI research. First publicly documented in 2021, its primary product is the Web Leacher Dataset, which has been used in the creation of datasets such as C4 and The Pile. The bot is designed to efficiently capture a broad cross-section of the internet while respecting publisher preferences when possible.

🌐 Technical Behavior

Webleacher employs a breadth-first crawl strategy, beginning from seed URLs in the Common Crawl index and expanding via extracted links. It typically issues requests at a rate of one per 10 seconds per domain, using persistent HTTP/1.1 connections and supporting HTTP/2. The crawler operates from a pool of IP addresses belonging to cloud providers such as AWS (EC2 instances) and DigitalOcean, with ranges documented in the project’s GitHub repository (https://github.com/webleacher/webleacher). It sends a User-Agent header and respects If-Modified-Since headers to reduce redundant downloads.

📋 robots.txt Compliance

According to the official Webleacher documentation (https://webleacher.net/robots), the bot fully honors Disallow directives found in robots.txt files. However, it does not honor Crawl-Delay directives, instead relying on its own configurable rate limits. Users can also block the bot via the X-Robots-Tag HTTP header. Empirical analysis from server logs confirms that Webleacher rarely violates explicit disallow rules.

🔍 Detection Indicators

The primary User-Agent string is "Mozilla/5.0 (compatible; Webleacher/1.0; +https://webleacher.net)" with variations for version increments. Additional identifying headers include From: [email protected] and a custom X-Crawler-Name: Webleacher. Behavioral fingerprints include a steady request interval of exactly 10 seconds per domain and the absence of JavaScript execution. The bot also includes a Via header indicating its proxy chain.

📊 Data Usage

The collected data is processed into a plain-text corpus, deduplicated, and made publicly available as the Web Leacher Dataset. This dataset has been used to train several notable language models, including GPT-NeoX and LLaMA, and is also employed by academic researchers for information retrieval benchmarks. The project publishes regular snapshots at https://data.webleacher.net.

⚙️ Rate Limiting Policy

Webleacher is rate-limited because its systematic crawling can impose a non-trivial load on origin servers, especially if multiple instances coordinate. Administrators are advised to implement threshold-based blocking (e.g., after 100 requests per minute) to preserve server responsiveness while still accommodating legitimate indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.