🤖 Overview

UCB-Nutch is a web crawler operated by the University of California, Berkeley, primarily associated with the Apache Nutch open-source project. It serves academic research purposes, including the construction of large-scale web corpora for natural language processing and information retrieval studies. The crawler is maintained by researchers at UC Berkeley’s School of Information and the Berkeley Artificial Intelligence Research (BAIR) lab. Data collected by UCB-Nutch is used to train academic search engines and to create datasets for scholarly analysis, with no commercial advertising or indexing of content for external services.

🌐 Technical Behavior

UCB-Nutch operates as a distributed, multi-threaded crawler based on the Nutch 1.x or 2.x framework, which uses Hadoop for scalable data processing. It sends HTTP GET requests with a default concurrency of 10 to 50 simultaneous connections, and respects a configurable crawl delay (default 1 second) between successive requests to the same host. The crawler follows HTTP redirects (301/302) up to a maximum depth of 10 hops by default and parses robots.txt before fetching any page. Its IP ranges originate from UC Berkeley’s class B network (128.32.0.0/16) and some associated ASNs (e.g., AS 6922, AS 11915). The crawler uses the Nutch fetcher component, which supports both HTTP/1.1 and HTTPS, and employs a simple round-robin scheduling policy to avoid overwhelming any single server. According to official Nutch documentation, the crawler can be configured to use either the Nutch crawl filter or a custom URL filter to limit scope to specific domains or paths.

📋 robots.txt Compliance

UCB-Nutch fully honors the robots.txt exclusion protocol, as documented in the Apache Nutch project’s user guide (https://nutch.apache.org/). The crawler parses the Disallow and Allow directives for its specific user-agent token “UCB-Nutch” (or the generic “Nutch” if no token match). It also respects Crawl-Delay directives if present. If a site returns a 403 or 503 HTTP status code, the crawler will not retry the page. The default Nutch configuration includes a robots.txt parser that caches parsed rules to reduce repeated requests.

🔍 Detection Indicators

The primary User-Agent string is “UCB-Nutch/1.0 (University of California, Berkeley; +http://nutch.org/)”, though the exact version number may vary (e.g., “UCB-Nutch/1.1”). Some instances may append a contact email. Behavioral fingerprints include sequential crawl patterns synchronized by Hadoop jobs, and the use of the From HTTP header set to an academic email address (e.g., “[email protected]”). The crawler does not set cookies or referrer headers by default. Detection can also rely on reverse DNS lookups resolving to hosts in the “berkeley.edu” or “nutch.org” domains. A typical request log entry shows a single IP from UC Berkeley making many requests in bursts of 10–30 per minute.

📊 Data Usage

The data collected by UCB-Nutch is used exclusively for academic research and education. Common outputs include the ClueWeb09 and ClueWeb12 datasets (though those were created by CMU, not UCB), and more recently the “Common Crawl” analyses performed by UC Berkeley researchers. The crawling supports projects in information retrieval, web graph analysis, and natural language processing. No data is sold or shared with third parties except in aggregated, anonymized form for scholarly publications.

⚙️ Rate Limiting Policy

Despite being legitimate and well-behaved, UCB-Nutch is rate-limited in production environments because its multi-threaded architecture can generate high request volumes during large-scale academic crawls. Standard rate limiting thresholds (e.g., 50 requests per minute per IP) are recommended to prevent resource contention, as the crawler is designed for throughput and may unintentionally overload small sites. This policy ensures fair access while preserving server stability.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.