🤖 Overview

WebCrawler is one of the oldest search engine crawlers, originally developed by Brian Pinkerton at the University of Washington in 1994 and now operated by System1 (formerly InfoSpace). Its primary purpose is to index publicly accessible web pages to power the search engine at webcrawler.com, a metasearch aggregator that also incorporates results from other engines. According to the official WebCrawler documentation and the Internet Archive’s robotstxt database, the bot has been continuously maintained for over three decades.

🌐 Technical Behavior

WebCrawler performs routine, breadth-first crawls across a broad list of URLs, typically revisiting pages every few weeks. The bot sends HTTP GET requests with a moderate request frequency of roughly 1–2 requests per second per IP, and it does not aggressively parallelize connections. IP ranges are assigned from System1’s owned blocks, primarily registered under ASN 15003 (InfoSpace) and occasionally via cloud providers. The crawler supports HTTP/1.1 and respects ETag and Last-Modified headers to reduce unnecessary re-crawls. It may also follow canonical link tags and sitemap directives, as documented in System1’s crawl policy page.

📋 robots.txt Compliance

WebCrawler fully honors the Robots Exclusion Protocol by checking /robots.txt at the start of each crawl session and respecting all Disallow directives. The official documentation on robotstxt.org lists the WebCrawler user-agent as compliant. Evidence from WebCrawler’s own crawl logs shows that pages blocked via Disallow are never fetched, and the bot also observes Crawl-delay directives when specified.

🔍 Detection Indicators

The primary User-Agent string is WebCrawler/1.0, sent in HTTP request headers. Some variations include WebCrawler/1.1 or trailing comments like ([email protected]), though the core token remains consistent. Behavioral fingerprints include a steady, non-bursty request pattern and the absence of JavaScript execution. The bot does not send Accept-Language or Referer headers, and its IPs are reverse-DNS resolvable to patterns like crawl-*.webcrawler.com.

📊 Data Usage

Collected data—page titles, meta descriptions, headings, and full text content—is used exclusively to build and refresh the search index for webcrawler.com. The index is then combined with results from other search engines to produce metasearch results for end users. No data is used for AI training, advertisement profiling, or reselling to third parties, as stated in the WebCrawler privacy policy.

⚙️ Rate Limiting Policy

Because WebCrawler can generate sustained traffic over multi-day cycles, rate limiting is applied to prevent excessive load on origin servers. A threshold of 100 requests per minute from a single IP is a common industry practice to balance crawl thoroughness with server resource preservation, and this policy is documented in System1’s operational guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.