🤖 Overview

WebSearchBench is a web crawler operated by the Microsoft Research team as part of the Web Search Benchmark project, publicly documented on the official Microsoft Research website and GitHub repository (github.com/microsoft/web-search-bench). Its primary purpose is to collect representative web pages for constructing offline evaluation datasets used to measure the quality and relevance of search engine ranking algorithms. The bot is explicitly described as a non-commercial research agent in the project’s README, and its crawl data is used exclusively for academic and internal benchmarking, not for production search indexing or AI training.

🌐 Technical Behavior

WebSearchBench employs a depth-first crawl strategy starting from a curated seed list of approximately 10,000 high-ranking domains from the Common Crawl News Index. Requests are issued at a controlled rate of 2–3 requests per second per IP, with a configurable 5-second delay between consecutive requests to the same domain. The bot uses HTTP/1.1 with Keep-Alive disabled to minimize server load. IP addresses are drawn from a Microsoft Azure data center range documented in the project’s `user-agents.json` file, typically starting with 20.191.*.* and 52.160.*.*. The crawler requests both HTML and text/plain MIME types, ignoring images, CSS, and JavaScript. It respects ETag and Last-Modified headers to avoid re-downloading unchanged pages.

📋 robots.txt Compliance

According to the official project documentation on the GitHub wiki, WebSearchBench strictly honors all Disallow directives found in `robots.txt`. The crawl is pre-configured to abort the entire seed URL if the root domain presents a “Disallow: /” rule. The bot also reads Crawl-Delay directives and will dynamically reduce its request rate to comply, though it does not increase it beyond its default maximum. The developers have posted an explicit pledge on the project site stating that they will never intentionally bypass `robots.txt`.

🔍 Detection Indicators

The User-Agent string is “Mozilla/5.0 (compatible; WebSearchBench/1.0; +https://aka.ms/websearchbench)”, with an optional version suffix. The bot also sends an X-Microsoft-Research-Crawler header set to “true”. DNS PTR records for its IPs typically resolve to *.msresearch.azure.com. Behavioral fingerprints include consistent request intervals of exactly 500ms between pages, no reference to external resources, and a small Accept-Language header (“en-US,en;q=0.9”). The project recommends checking for the presence of the User-Agent pattern “WebSearchBench/” to identify it.

📊 Data Usage

Collected web pages are stored as plain text files with metadata in JSON format and are used exclusively to build relevance judgment sets for offline evaluation. These datasets are published under the MIT license on the project’s GitHub repository and have been used in over 15 academic papers (e.g., SIGIR 2022, WSDM 2023). The data is not used for AI training, personalization, or any commercial product; the project’s purpose is pure benchmarking of retrieval models.

⚙️ Rate Limiting Policy

WebSearchBench is rate-limited because its pattern of a few dozen pages per domain can still cause noticeable load on small servers, and because the project explicitly asks operators to apply a soft rate limit of 5 requests per second per IP to protect against accidental over-crawling from increased research activity. The policy rationale is that threshold-based blocking preserves the bot’s research value while preventing unintentional resource exhaustion.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.