🤖 Overview

Gulliver is the name of the web crawler operated by the Common Crawl Foundation, a nonprofit that maintains a free, open repository of web crawl data. First deployed in 2008, Gulliver is designed to periodically crawl the entire public web, collecting raw HTML pages and metadata for inclusion in the Common Crawl dataset, which is used by researchers, companies, and hobbyists for AI training, language models, and large‑scale text analysis. The crawler is open‑source and built on Apache Nutch, with its code hosted on the Common Crawl GitHub repository (github.com/commoncrawl).

🌐 Technical Behavior

Gulliver performs broad, breadth‑first crawls, starting from a seed list of high‑authority domains and following links to cover billions of pages per month. The crawler respects standard HTTP protocols, uses conditional GET requests (If‑Modified‑Since and ETag headers) to avoid re‑downloading unchanged content, and typically sends requests at a moderate rate to minimize server impact — often on the order of a few requests per second per IP. As of 2025, Common Crawl publishes monthly snapshots; each snapshot typically contains 5–10 petabytes of uncompressed data. IP addresses are predominantly assigned from Amazon Web Services (AWS) EC2 ranges, particularly us‑east‑1, us‑west‑2, and eu‑west‑1, and may rotate frequently. The crawler uses a custom HTTP client based on Java’s URLConnection and adheres to HTTP/1.1 keep‑alive connections.

📋 robots.txt Compliance

Common Crawl explicitly states that Gulliver fully respects robots.txt directives. Before crawling a domain, the crawler fetches and caches the robots.txt file, obeying both Disallow and Crawl‑delay instructions. Documentation on the Common Crawl wiki (commoncrawl.org/faq) confirms that the project automatically excludes any pages blocked by robots.txt, and site owners can opt out entirely by disallowing the CCBot user‑agent string (which Gulliver uses).

🔍 Detection Indicators

Gulliver identifies itself with the User‑Agent string CCBot/2.0 (https://commoncrawl.org/faq/) in all HTTP requests. Additionally, the crawler includes a custom header From: googlebot(at)commoncrawl.org (though this may vary across versions). Reverse DNS lookups on crawling IPs typically resolve to hostnames ending in .compute.amazonaws.com. The crawler does not mimic browser fingerprints; it sends a minimal set of headers and lacks JavaScript execution capability.

📊 Data Usage

The data collected by Gulliver is publicly released as the Common Crawl dataset, hosted on Amazon S3 and available for free download. This dataset feeds countless AI training pipelines — including large language models like GPT and BLOOM — academic research in information retrieval, natural language processing, and web science, as well as commercial search‑engine optimization tools. Common Crawl explicitly permits any use under its terms, making its corpus one of the largest and most widely used public web archives.

⚙️ Rate Limiting Policy

Although Gulliver is a legitimate, non‑malicious crawler, its ability to generate high volumes of requests by rotating across many AWS IPs means it can still cause performance strain on under‑provisioned servers. Rate‑limiting to a threshold (e.g., 5 requests per second per IP or blocking after a burst of 100 requests in 10 seconds) is a prudent operational practice to protect application availability while still allowing the crawler’s valuable data collection to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.