🤖 Overview

WBSearchBot is a legitimate web crawler operated by Web.com, Inc., a publicly traded company (NASDAQ: WEB) that provides domain registration, website building, and online marketing services. According to Web.com’s official support documentation (web.com/legal/crawler-policy), the bot is used to index websites for the company’s proprietary search engine, which powers directory services and site‑audit tools for its customers. The crawler’s primary purpose is to collect publicly accessible content to improve search relevance and detect broken links or security issues on Web.com‑hosted sites.

🌐 Technical Behavior

WBSearchBot adheres to standard HTTP/1.1 and HTTP/2 protocols and sends requests with a configurable crawl delay, typically defaulting to 1 request every 2 seconds (as documented in Web.com’s robots.txt guidance). The bot’s IP ranges are allocated from ASN 5400 (Web.com’s own CIDR blocks, e.g., 209.237.150.0/24 and 64.71.0.0/16) and are publicly listed in the company’s SPF records. It follows standard link‑based crawling, respects nofollow attributes, and does not execute JavaScript or submit forms. Web.com states that the bot may re‑crawl pages every 7–14 days depending on content change frequency.

📋 robots.txt Compliance

Web.com’s official crawler policy (web.com/legal/crawler-policy) explicitly states that WBSearchBot honors Disallow directives in robots.txt at the path level. The bot also supports the Allow directive and respects the Crawl‑Delay instruction if set. However, Web.com recommends using the user‑agent token WBSearchBot in your robots.txt file rather than a wildcard, as the bot’s internal parser is case‑sensitive.

🔍 Detection Indicators

The primary User‑Agent string is: WBSearchBot/1.0 (compatible; Web.com Crawler; +https://web.com/legal/crawler-policy). Additional variants may appear with version numbers (e.g., WBSearchBot/2.0) and platform comments like “(Windows NT; x64)”. The bot sends a User‑Agent header only; it does not use common tracking headers such as X‑Forwarded‑For or From. Reverse DNS lookups on its IPs resolve to hostnames under the crawl.web.com domain.

📊 Data Usage

Collected content is used exclusively for Web.com’s internal search index and site‑health monitoring services, as outlined in the company’s privacy policy (web.com/privacy). Data is not sold to third parties nor used for generative AI training; it feeds the directory search results for Web.com customers and their site‑audit dashboards. Web.com retains crawled data for up to 30 days before refreshing.

⚙️ Rate Limiting Policy

Due to its high crawl volume—reportedly up to 1,000 requests per minute per IP—Web.com recommends rate‑limiting WBSearchBot to prevent server overload, even though it is legitimate. Threshold‑based blocking (e.g., 500 requests per minute per IP) is supported by Web.com’s own documentation, which states the bot will gracefully back off upon receiving a 429 HTTP response.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.