🤖 Overview

libcrawl is a non‑profit web crawler operated by libcrawl.org, established in 2019 to collect open web content for academic AI training and linguistic research. Its dataset feeds into large‑scale language model projects and is available under a Creative Commons license.

🌐 Technical Behavior

libcrawl uses a distributed Scrapy architecture, sending 1–5 requests per second per IP with a default 2‑second crawl delay. It identifies via User‑Agent libcrawl/1.0 (+https://libcrawl.org/bot) and uses HTTP/1.1 with ETag caching. The crawler operates from IPv4 addresses in ASN 12345 and IPv6 prefix 2001:db8::/32. It respects noindex metas and nofollow links, and does not execute JavaScript.

📋 robots.txt Compliance

libcrawl fully respects robots.txt directives. It caches robots.txt for 24 hours and never visits Disallowed paths, as documented at libcrawl.org/docs/robots. No violations have been reported.

🔍 Detection Indicators

The primary User‑Agent string is libcrawl/1.0 or libcrawl/2.0. Additional headers include X-Crawl-ID and From: [email protected]. Its IPs reverse‑resolve to crawler-*.libcrawl.org. The bot may be identified by its steady request rate and lack of randomness in intervals.

📊 Data Usage

Text and metadata extracted from crawled pages are used to train transformer models and benchmark information retrieval systems. libcrawl releases annual WARC snapshots on Zenodo and BitTorrent for non‑commercial research use.

⚙️ Rate Limiting Policy

Despite its good behavior, libcrawl’s sustained crawl rate can strain small sites. Rate‑limiting at 10 requests/second per IP or when 404 errors exceed 5% is recommended to protect server resources while allowing the crawler to complete its mission.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.