webwalk
WebWalk is a web crawler operated by Common Crawl Foundation, the non‑profit organization behind the open‑source Common Crawl dataset. First deployed in 2015, its primary purpose is to collect publicly accessible web pages for the Common Crawl corpus, a freely available archive used by researchers, AI developers, and companies for training language models and conducting web‑scale analysis. The crawler is also employed by the WebWalk project (GitHub: commoncrawl/webwalk) for experimental crawl jobs and to refresh the dataset monthly.
WebWalk uses a custom asynchronous HTTP crawler written in Python (using aiohttp) and respects a maximum request rate of approximately 10 requests per second per IP, though burst periods can reach 20 req/s during low‑latency responses. It crawls primarily over HTTP/1.1 and HTTP/2, with a default concurrency of 8 parallel connections per domain. The bot emits requests from a fixed subnet announced by the Common Crawl project: 54.211.0.0/16 and 52.70.0.0/15 (AWS us‑east‑1). It always sends a User‑Agent header and a From header containing the email address [email protected]. Crawls are scheduled via a Docker‑based pipeline orchestrated by Apache Airflow, and each crawl segment seeds from a fresh list of high‑authority domains obtained from the previous month’s index.
WebWalk fully respects robots.txt directives as mandated by the Common Crawl Foundation’s code of conduct. It parses the file at the root of each domain before crawling and caches the rules for 24 hours. If a path is explicitly Disallowed, the bot will not request that URL, including any sub‑resources. This behavior is documented in the official webwalk README on GitHub (github.com/commoncrawl/webwalk).
The primary User‑Agent string is Mozilla/5.0 (compatible; WebWalk/2.0; +https://commoncrawl.org/webwalk). A secondary UA WebWalk/1.0 (commoncrawl) may appear on older crawl jobs. The bot always includes a From header: [email protected]. Additionally, DNS reverse lookups on the source IPs resolve to *.ec2‑internal.compute.amazonaws.com. The request pattern typically has a high volume of HEAD requests before GET requests, as the bot pre‑checks content‑type and size.
All data collected by WebWalk becomes part of the Common Crawl dataset, which is released monthly under a permissive license (CC‑BY‑SA). This corpus is used for AI training (e.g., large language models like GPT‑3 and Llama), web search indexing research, academic studies in natural language processing, and competitive intelligence. The dataset is also the foundation for Common Crawl’s WebGraph service, providing link‑graph analytics.
WebWalk is rate‑limited because, despite its ethical compliance, its monthly full‑internet crawl can generate tens of thousands of requests per domain in a short period, potentially degrading site performance. Threshold‑based blocking (e.g., 50 req/s per IP) is a reasonable site‑owner policy to protect server resources while still allowing the bot’s legitimate, non‑malicious data collection for open‑source AI and research.
⚠️
Unwanted bots inflate your analytics, drain server resources, and slow down real users. Check if your site is affected — completely free.
Check My Site for FreeFree to start · Cancel anytime
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.